Full text of "Rethinking Quality Assurance For Higher Educarion in Brazil"

See other formats

Reviews of National Policies for Education

Rethinking Quality Assurance
for Higher Education in Brazil

aan I eee
Xn 4
YS

————
EE
——
ae
———
ee
—
=
——
—
——
=
Oo
reps

Reviews of National Policies for Education

Rethinking Quality
Assurance for Higher
Education in Brazil

@)) OECD

BETTER POLICIES FOR BETTER LIVES

This work is published under the responsibility of the Secretary-General of the OECD. The
opinions expressed and arguments employed herein do not necessarily reflect the official
views of OECD member countries.

This document, as well as any data and any map included herein, are without prejudice
to the status of or sovereignty over any territory, to the delimitation of international
frontiers and boundaries and to the name of any territory, city or area.

Please cite this publication as:

OECD (2018), Rethinking Quality Assurance for Higher Education in Brazil, Reviews of National Policies for
Education, OECD Publishing, Paris.

https://doi.org/10.1787/9789264309050-en

ISBN 978-92-64-30904-3 (print)
ISBN 978-92-64-30905-0 (pdf)

Series: Reviews of National Policies for Education
ISSN 1563-4914 (print)
ISSN 1990-0198 (online)

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of
such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in
the West Bank under the terms of international law.

Photo credits: Cover © eabff/Shutterstock.com

Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.
© OECD 2018

You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and
multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable
acknowledgement of OECD as source and copyright owner is given. All requests for public or commercial use and translation rights should
be submitted to rights@oecd.org. Requests for permission to photocopy portions of this material for public or commercial use shall be
addressed directly to the Copyright Clearance Center (CCC) at info@copyright.com or the Centre francais d’exploitation du droit de copie
(CFC) at contact@cfcopies.com.

FOREWORD | 3

Foreword

Brazil has well-established systems to assess, monitor and assure the quality of learning
and teaching in private and public higher education institutions in the federal higher
education system. Participation in the external quality assurance system is mandatory for
private and federal public higher education institutions, which together account for over
90% of undergraduate enrolment in Brazil. The procedures used to assess institutions and
undergraduate programmes are wide-ranging, encompassing institutional self-evaluation,
peer-review visits, programme-level quality indicators and large-scale student performance
assessment. A separate, comprehensive system of external peer review is used to assure the
quality of academic postgraduate education throughout the country.

At the undergraduate level, the National System of Higher Education Evaluation (SINAES)
and related regulatory and supervisory systems have been operating, with some
modifications, since 2004. SINAES has become a fixture of the Brazilian higher education
landscape, but has, itself, never been subject to a systematic assessment of its relevance,
effectiveness and efficiency. The system for evaluating academic postgraduate
programmes, run by the Foundation for the Coordination of Improvement of Higher
Education Personnel (CAPES) has been operating since 1998. It is well respected in the
academic community in Brazil, but as postgraduate education continues to expand, there
are questions about the sustainability of its intensive peer-review processes.

Against this backdrop, the Ministry of Education (MEC), the National Commission for
Evaluation of Higher Education (CONAES) and CAPES asked the OECD to undertake an
independent review of federal quality assurance policies for higher education. The terms of
reference for this review called on the review team, composed of international experts and
OECD staff, to assess the relevance, effectiveness and efficiency of the procedures in place.
The team was asked to consider the effectiveness of the systems in ensuring minimum
quality standards, providing differentiated measurement of quality and promoting
improvement of quality and quality-oriented practices in higher education institutions. The
team’s findings and recommendations are presented in this report.

I hope this report will support Brazil in its efforts to promote quality in its higher education
system through effective systems of quality assurance. The OECD stands ready to help
Brazil in these efforts.

Jee eect

Andreas Schleicher

Director for Education and Skills and Special Advisor
on Education Policy to the Secretary General
OECD

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

ACKNOWLEDEMENTS | 5

Acknowledgements

This review is part of the OECD’s efforts to support countries to enhance the relevance,
effectiveness and efficiency of their education policies, coordinated by the Policy Advice
and Implementation Division within the Directorate for Education and Skills (EDU). The
review was coordinated by Thomas Weko (Senior Analyst, OECD), and preparation of this
report was coordinated by Simon Roy (Analyst, OECD). The review team (see Annex A)
consisted of Maria José Lemaitre (Executive Director, Centro Interuniversitario de
Desarrollo, Santiago, Chile); Pedro N. Teixeira (Associate Professor, University of Porto,
and Director of the Centre of Research on Higher Education Policy, CIPES); Thomas Weko
(Senior Analyst, OECD) and Simon Roy (Analyst, OECD). Simon Roy and Thomas Weko
authored the report, with the support of OECD consultant Manuela Fitzpatrick, drawing
upon input from Maria José Lemaitre and Pedro N. Teixeira.

The National Commission for Evaluation of Higher Education (CONAES) and the Ministry
of Education (MEC) initiated the review. It benefitted from the support of CONAES
members Professor Simon Schwartzman and Professor Renato Hyuda Pedrosa. Professor
Elizabeth Balbachevsky, from the University of Sao Paulo, developed a country
background report to inform the review team. Within MEC, the review was supported by
the leadership of Maria Helena Guimaraes Castro (Vice-Minister), Henrique Sartori de
Almeida Prado (Secretary for Regulation and Supervision of Higher Education, SERES)
and Maria Inés Fini (President, Anisio Teixeira National Institute for Educational Studies
and Research, INEP). Both INEP and MEC provided data and documentation to inform the
analysis, the former under the leadership of Mariangela Abrao (Director of Higher
Education Evaluation, INEP). The review team is indebted to Ricardo Corréa Coelho, who
acted as the principal point of contact for the review in MEC and to Leandro Gomes
Cardoso and Manoela Vilela Aratjo Resende, who provided invaluable logistical support.

The team met with a wide range of government officials and higher education stakeholders
during a review visit to Brazil in March 2018, and is grateful for their time and input. They
included members of MEC staff responsible for higher education; key staff from INEP;
federal, state, and private bodies responsible for supporting postgraduate education and
administrators, academic staff, and students from private and public higher education
institutions located in Brasilia, Recife, and SAo Paulo (see Annex B). Thomas Weko and
Simon Roy presented preliminary findings and recommendations from the review to
stakeholders in the higher education quality assurance system at a workshop held in Brasilia
on 27 August 2018. The review team is grateful for the feedback received during and after
this event, that has helped to sharpen the final version of the report.

Key planning and operational support for the project was provided by Manuela Fitzpatrick
(OECD, consultant), and administrative support was provided by Jonathan Wright (OECD,
Project Assistant), who also prepared the report for publication. The review team is grateful
for the feedback and support received from Andreas Schleicher (Director, Directorate for
Education and Skills, OECD) and Paulo Santiago (Division Head, Policy Advice and

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

6 | ACKNOWLEGEMENTS

Implementation Division, EDU, OECD) and Elizabeth Fordham (Global Relations
Advisor, EDU, OECD).

While the OECD review team benefitted greatly from many discussions with a wide range
of Brazilian stakeholders, as well as documents, data, and a country background report, any
errors or misinterpretations in this report are the responsibility of the review team.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

TABLE OF CONTENTS | 7

Table of contents

FOLOWOLG .......ssscsscssscsssecsscsssssesssnsessssssecesssnsssesssnssessecsscssesenacsasesssncssecssssscssansensecsscsscssosesacsassssssessessenseess 3
ACKNOWIEASEMENUS sicccnssiscsasvescsscccnssseastoacescsussseanseecssccuessessedessncseutassosendsosessvecsednsdassvesostessenenddonsssvevescoeus 5
Abbreviations and acronym ............sccccscssccsccccccsccccssccssssccccsescesccesssscesssccecscscescesssscessscssccecssescesssssesees 11
EXe€CUtEVE SUMMATY ........ssccscesssscversvcssecsscssescecssnssessssscesssssscsecsensscsscsscssassnassasessssesseessssuessacsonssesscsseses 13
Regulating “market entry”: ensuring providers of higher education meet high standards, while
streamlining quality assurance ProCeCUres ...........eecceseceseeseeeeceseeseeecsecaeeeceesecaaeeaeesecaesaeseeeeaecaeeeneeaes 13
Ongoing quality assurance of undergraduate programmes: improving measurement of quality and
better targeting Of TESOUNCES 2 sssccse) secs catecosccesdbesccvsastake cand wescedaneseecavbedecesossedesavsucssabevatesuaaeveccvasesmiecautecees 14
Quality in postgraduate programmes: fine-tuning existing practice and planning for the future ...... 15
Quality higher education institutions: increasing focus on internal quality assurance and the role
of external institutional TEVICWS ..........ececeeeceescesecseeeeeesecseesseeseceaesaeesecsaecaeeenesecaaeeeeesecnaesaeseeceaecaeeeneease 15
Governance of the quality assurance system: ensuring greater transparency, improved steering
and engagement of Sector OrgamisatiONS ...........cccecccessceesceeeeeeseeeseeeseecseecseecsaecueceaeceaecnseeeseeseeeseeeeeneesgs 15
1. Assessment and reCOMMENAAtiONG..............ccsscccsccsscessccssscesscesssssssssscsssssssssesssesssesssessssssesssssssoeees 17
1.1... FOCUS-OF THIS: CHAPTER 5.1. cisccessdecsacadacdieessutbantesunsaasdabevece tandeseshdhdcens Stuaste stan ation Gegheeogutebeasoueeeecessaeeda 18
1.2. External quality assurance in Brazil: evaluation, regulation and superviSion............::cseeeeeeeee 18
1.3. Regulating market entry: new institutions and undergraduate programmMes............:ecceseeeereeeee 20
1.4. Assuring and promoting quality for existing undergraduate programmes. ..........:.cceecceseeeeeeeeees 26
1.5. Assuring the quality of postgraduate education ..........cccesccesceseceseceeseeeeeeeseeeseeeseecseecnaeenseenseenaeens 35
1.6. Assuring the quality of higher education institutions .0........cececeeeeeeeeceeeeseeeeceaeeeeeeeeeaeeaeeeneeaee 39
1.7. Governance of external quality ASSUTANCE 000... eee ceseesteceseceseceseceseceeeeeeeeeeeeeeseeeseeesaecsaecaeenaeenaeens 42
RGLETOMCES 1555, 535i 555 selebetedecasesdaeciseaednstdesecadaliecdodhesacedevas oetostas oeavasg nsdeubasoauesapadueagandaneaasousevaa suagswanssedee 47
2. Scope of the review and analytical fraMmework.............scscccsscssccsscssscssscsssscssscesscssscsesssesssessessseees 49
2 |i: FOCUS-OF UNIS CHAPUGE Serscecscenss2c pivennee.ahdudeteivenctv chad Mooctwoneacestde tacts es neiveautvascesseeesteatn dads eswerearnz 50
2.2. External quality assurance of higher education in Brazil... cc cseecceseeseeseetecneeeeeeecneeeaeeeeeeaeeas 50
2.3. Quality assurance in higher education internationally: developments and challenges ............... 53
2.4. An analytical framework for the review .........cccccsceseceseceseceeceeceeeeseeeesneceseesseecseecseeeaecnaeenseenseees 58
IRGLSTONCES -..5:2522.c55ihcscesttbecdeecaneeel sagen sateuaces seaneestaaaceckatesaeass caatans baa daaesdasdaaeaenstenstaaaee nae 63
3. Higher education in BraZil............scscsccsscssssssssssssssssssessscecssssessecsscssesssassassssssessesssssessecssnssessecsseses 65
3.1 Focus:Of this: Chapter veces. ccccesatsnsenets dosteesteives sued ccxesvantdunconndadues tobdeiersdavarvetucdensts nwesrecteudesltaduaeeisteids 66
3.2. The socio-economic context for higher education 1n BraZil ..........cccesceseceseceseceeeeeeeeeeeeeseeseeeeses 66
3.3. Governance arrangements, funding and key public policies for higher education .................0. 69
3.4. Higher education provision in Brazil .........cccccesscsssceseeeseeeeeeeecseeceecaecaeceaecesecnaeeeeeseeeseeeeeeeeaes 76
3.5. Main trends in participation, equity and OULCOMES ..........ccscecsseesseesseesseceseceseceseceseeeeeseeeeseesneeeses 81
RGLSTONCES viii a ciensdiegedebenaedsennsiseedne sexs uae ed dcteb sie desteey caves oe idea Seas cade ate aeiaatg dane a eda ate eane deateannetaees 87

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

8 | TABLE OF CONTENTS

4. Market entry: New institutions and undergraduate programMes..............ccccsrcsscsssersseesseoees 91
AS FOCUS: OF Chis Chapter tscisssevteatcscsaubacasatea Soscdectics atest bisstebaes cagesbassacehvecaudestha sadebacs codes hatisaes veceedseshieeds 92
4.2. Strengths and weaknesses of the current SYSteM .........:.ccccesccessceeseeeeeeeeeeeseeeseeesseceecsaecnaeenseeeseens 93
4.3. Key recommendations «.22..22h2vissipectatist ase eaes ev iieciadive ag eel ciple ade, chad ateeel gb ehh vetde die 104
ROPETCNCES 2A sets cecs ch dt eces es ones ios iti nesiad sah dacens Maa teet ena tecks vale sateacaca au Sesel She tena tbe s tecas dnedvetintassaast 107

5. Assuring and promoting quality for existing undergraduate programmMecs...............ccesseeeeees 111
§:1.,Focus:of this:chapter.::.,2.2045, ,sveis.:asisiecslovige woathsae tide gdeageisld i ohaaiate lashed pian dieser 112
5.2. Strengths and weaknesses of the Current SYSteM ..........cccccssccssecsteceeceeceseceeeceeeeeeseeeseeeseeeseeenaes 113
5:3; Key TECOMMENCALIONS |. 55 Ssnccaccesc eves cea shes aien hSSs Segav nc eaagon Vol Seg ahaa woedaa Ses twage ad volgen te lead wbis eben cabot sen senens 128
BRS) £51 0 hs an a tn ee 133

6. Assuring the quality of postgraduate education. ..............ccsscssscssccssssssccssscssscssscsssssssscsscsssoesees 137
6.4: Foctis: of this: chapter chic. ides vet cs bacco vawsicdeoas Lets va thedaved ss Mea teaivedats devak a wkeded ceded viatecetabecetiees 138
6.2. Strengths and weaknesses of the Current SYSteM ........:.ccccesecssecsseceteceeceeeeeeeeeeeeeseesseeeseeeeeesaes 139
6.3. Key recommendations concerning CAPES evaluations ..........:ccccesccesscessceeeceeeceeseeeseeeneeesseesaes 151
References 22252 dscsssacteceg ho csech peas cenb pedo ad caieoas hy ade saegiaa bay dtesadsaeeia bin ddvebaptadeas eudedaageding he dicosapea len ares 155

7. Assuring the quality of higher education institutions ................sccssccsscsssscsscsscsssssscsscsssseesees 157
7.1. Focus of this chapter .........ccccccccssccssecssscssscesecesecessececeeeseeeseecsaecsaecsaecaecesecsseeseeeseseseseseaeeeeeeeeesaes 158
7.2. Strengths and weaknesses of the Current SYSteM .........:ccccssecsseceseceteceseceeceeeeseeceeseeeseeeseeeseeesaes 160
7.3. Key recommendations concerning institutional revieW..........ccccesccesecesscessceeeceeeeesseeeseeeeeeeeesaes 164
REPET ENCES Soest yseece tac zieeca Sasa ston dues eats cor coage cut voseassh uses bees tone dase natu onan ot ate hatesnantsel cashout s oueaveal oa tenons 167

8. Governance Of external quality ASSUTANCE ..............sccssccssccscrsensscnecsesssesssescsesscssssssescsesceesseesees 169
Bil Focus: ofthis Chapter oss sscceissetioves oe wonsealineel ott lavend deeviaceasesevenl nesianghe tes vehiveeshemeveene nen ates 170
8.2. Strengths and weaknesses of the Current SYStM ..........ccccecccessceesceeeceeeeeeseeeseeeseecsaeesaecnseenseenaeens 170
8.3. Key recommendations CONCerMING ZOVEFMANCE ........eeceeccesscessceeeceeeeeeseeeseeescecseecseecsuecsseenseeeaeees 172
RGLeTeN Ces ite th tt tet aot, ate tte tateMrst tts, Mutts ok talatec treet scuettauhs cate k ts meet teat A Aa laat aes 175

Annex A. Review team ............ccssccsssssessscssccsesssscssecssscsssssssscsescsssnssnsssnessnessssssenssesssssessssssssssssesooese 177

Annex B. Schedule for the review team’s mission tO Brazil .............ssccsscssscssscsssccssscssesssesssscoees 178

Tables

Table 2.1. Quality indicators: objectives, inputs, processes, outputs and OULCOMES ..........cceeteeteeeteees 57

Table 2.2. Factors in effective quality assurance SYStOMS ..........:cccccesceesceeeceeeceeseeeseeeseecseeeaecteenseenaeens 61

Table 3.1. Number of tertiary education institutions, by type and sector (2016) ......ccceseeseeeteeeteeeteees 77

Table 3.2. Undergraduate enrolment, total and share ...........cccecccescessceseceeeeeeeeeeseceseeeseecseeesaeesseceseenaeees 78

Table 3.3. Teaching staff by contract, sector (2016) ......cececsccsseceseceseceseceeeeeeeeeeseesseeeseecseeesaecnseceseenseees 81

Table 4.1. Regulatory acts by SERES in 2017 .o....ccccccccsccssecsseceseceseceecseeceeeeesseceseeeseecsaeessecsseenseenaeens 93

Table 4.2. Indicators used for on-site inspections for institutional accreditation ...........ceeeeeeeeseeeeeeee 95

Table 4.3. Indicators used for authorisation (QUtOIZAGGO) ...cccececesscessceseceseceseceeeeeeeceeeeeeaeesseeeseeeaeesaes 101

Table 4.4. Indicators used for recognition (reCONNECIMENLO) .....cccesecessceseceseceesceeeceeeeeeseeeseeeseeeseeesaeesaes 101

Table 5. ENADE cycle fess sorciiebascdeariitecsa Heasetianitias eboedhaeadioes died aha beads detained bed ata ees 117

Table 5.2. Indicators used to calculate the CPC ........cccccccseessecsseesecetecesecesecesecseeseeeseseseaeceseeeseeeaeenaes 123

Table 6.1. Criteria for evaluation of new postgraduate course proposals .........cccccceeseesseetseeteeesseesees 141

Table 6.2. Periodic programme evaluation: criteria for academic programmMes............:eecceseeseeeeeeteees 147

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

TABLE OF CONTENTS | 9

Table 7.1. Duration of accreditation and re-accreditation ..........cceecceseeeceescesececeeceeseceeeeeeeeceaeeaeeereeaeeas 162
Table 7.2. Number and weight of indicators for institutional re-accreditation and change of
TUS TUEUIOD AL STATUS aosess ns sub ate ce nonce serancinsset ecw ane pares Risaratenastuvousls reac saauucusaiecahNen eta ie sautiuatsd uoeoassounaeas 163
Table 7.3. Type of indicators used in the re-accreditation Process .........eccesceseeseeseceeeeeeeeceseeseeeeeeaeeas 163
Figures
Figure 2.1. Regulation, evaluation and supervision in Brazilian higher education ..............:eseeeeeeeee 52
Figure 2.2. External quality assurance system for higher education in Brazil .....0.....ceceseeseeseeeeeereeees 59
Figure 3.1. GDP growth and unemployment in Brazil... cee eeceeseecceseeeeeeceseceeeeceesecaeeaeeeeceaeeaeeeneeaes 67
Figure: 32: Income: [me qualtty vec: c.noc. deen svacectcavecsetvea shade to soenes cebcodeved dose ovuhevagde vi eouscavrsagnaenstoussaveacsutedvenns 68
Figure 3.3. Current governance arrangements for the federal higher education system in Brazil
(November 2018) cesses odeeeeiias Saccntiede. alveecnt ade k ete ave idedevn teas een vache dete adeea ode taut 70
Figure 3.4. Total expenditure on educational institutions as a percentage of GDP (2015)... 72

Figure 3.5. General government and private expenditure per full-time tertiary student in public
higher education institutions, (ISCED 2011 levels 5 to 8), including research and development

C2O1D Ves sccriesh ecues sided oevat trata atavia vite tae rindegk ona aetctahcase aaa distr aeeaihan Ramicsamad as utenauel ach ninangsenaetes 73
Figure 3.6. Number of stricto sensu master’s and doctoral programmes, 1996-2014.........ceseeseeeeees 79
Figure 3.7. Share of professors by educational attainment, by SeCtOF .........eceececeeseeeeceeeeeeeceseeaeeeneeaes 80
Figure 3.8. Educational attainment of 25-34 year-olds (2017).......c.cesccesseessceseeeeseeeteeeseecseecnseenseenseenseens 81
Figure 3.9. Enrolment in undergraduate programmes ...........c:ccssccesscessceseceeeceeeeeeseeeseeeseecsaeesaecsaecnaeeeaeees 82
Figure 3.10. Enrolment in distance education programmes (2009-15), by S€CtOL.......cccecceseeteeeteeeteees 83
Figure 3.11. Share of enrolment in higher education institutions, by income quartile and sector ........ 84
Figure 3.12. Share of students in higher education by mode of study and age (2015)... eeseeteeeteees 85
Figure 3.13. Relative earnings of adults, by educational attainment (2016) 0.00... eeeeceeseeeceeeeneeeneeees 86
Figure 5.1. The cycle of evaluation and regulation for undergraduate programmes...........::c:ccceeees 113
Figure 5.2. Standardisation of ENADE SCOPES ...........ccceseesseseceseeseeecesesaeeeeceaecaeeeceeaecaaeeaeesecaeeaeeeeeaeeas 118

Boxes
Box 3.1. Higher education goals in the National Education Plan 2014-2024 oo... ecceceseseeceseeneeeeeeees 75
Box 4.1. Sanctions used by SERES in supervision of (private) higher education ............ccccssesseeeees 100

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

ABBREVIATIONS AND ACRONYMS. | 11

Abbreviations and acronyms

APCN Avaliacgao de Propostas de Cursos Novos (CAPES Evaluation of Proposals for
New Courses)

CAPES Coordenagado de Aperfeigoamento de Pessoal de Nivel Superior
(Foundation for the Coordination of Improvement of Higher Education Personnel)

CC Conceito de Curso (Course Score)

CEFET Centro Federal de Educagdo Tecnologica (Federal Technological Education
Centre)

CENEVAL — Centro Nacional de Evaluacion para la Educacion Superior (National
Centre for Higher Education Evaluation, Mexico)

CI Conceito Institucional (Institutional Score)
CNE_ Conselho Nacional de Educagdao (National Education Council)

CNPq Conselho Nacional de Desenvolvimento Cientifico e Tecnoldgico (National
Council for Scientific and Technological Development)

CONAES Comissao Nacional de Avaliagao da Educacgao Superior (National
Commission for Evaluation of Higher Education)

CPA Comissao Propria de Avaliagao (Internal Evaluation Commission)
CPC Conceito Preliminar de Curso (Preliminary Course Score)

CPF numero do Cadastro de Pessoas Fisicas (Population Register Number)
CST Curso superior de tecnologia (Advanced Technology Programme)

CTC-ES Conselho Técnico e Cientifico da Educagao Superior (Technical and
Scientific Council for Higher Education)

DAES Diretoria de Avaliagdo da Educagao Superior (Directorate for Evaluation of
Higher Education — part of INEP)

DCN _ Diretrizes Curriculares Nacionais (National Curriculum Guidelines)

EGEL Exdamenes Generales de Egreso de la Licenciatura (General Examination for
Graduating Bachelor’s Students, Mexico)

ENADE Exame Nacional de Desempenho de Estudantes (National Examination of
Student Performance)

ENEM Exame Nacional do Ensino Médio (National Examination of Upper Secondary
Education)

HEI Higher Education Institution

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

12 | ABBREVIATIONS AND ACCRONYMS

INSAES Instituto Nacional de Supervisdo e Avaliagdo da Educagao Superior
(National Institute for Supervision and Evaluation of Higher Education — proposed, but
never created)

ISCED International Standard Classification of Education

IDD — Indicador de Diferenga entre os Desempenhos Observado e Esperado (Indicator of
difference between observed and expected performance)

IGC Indice Geral de Cursos (General Course Index)

INEP Jnstituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira (Anisio
Teixeira National Institute for Educational Studies and Research)

MBA Master of Business Administration
MEC Ministério da Educagdao (Ministry of Education)

NDE Micleo Docente Estruturante (Structuring Teaching Core: group of teaching staff
responsible for designing and maintaining programmes)

OAB_ Ordem dos Advogados do Brasil (Brazilian Bar Association)

OECD Organisation for Economic Cooperation and Development

PDI = Plano de Desenvolvimento Institucional (Institutional Development Plan)
PhD Philosophiae doctor (Doctor of Philosophy)

PNE Plano Nacional de Educagao (National Education Plan)

PNPC Programa Nacional de Posgrados de Calidad (National Programme for Quality
Postgraduate education, Mexico)

PPC Projeto Pedagogico do Curso (Programme Pedagogical Project)

SERES Secretaria de Regulacgao e Supervisao da Educagao Superior (Secretariat
for Regulation and Supervision of Higher Education)

SESu Secretaria de Educagao Superior (Secretariat for Higher Education)

SETECSecretaria de Educagdao Profissional e Tecnologica (Secretariat for Professional
and Technological Education)

SINAES Sistema Nacional de Avaliagao da Educagao Superior (National System of
Higher Education Evaluation)

SNPG Sistema Nacional de Pos-graduacdao (National System of Postgraduate Education)

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

EXECUTIVE SUMMARY | 13

Executive summary

Brazil has well-established systems to assess, monitor and assure the quality of learning
and teaching in private and public higher education providers in the federal higher
education system. The Ministry of Education (MEC) makes regulatory decisions about
accreditation of institutions and authorisation and recognition of undergraduate
programmes based on the results of evaluations coordinated by the Anisio Teixeira
National Institute for Educational Studies and Research (INEP). The Foundation for the
Coordination of Improvement of Higher Education Personnel (CAPES) implements a
separate system of quality assurance for academic postgraduate programmes.

The external quality processes for higher education institutions (HEIs) and undergraduate
programmes are mandatory and apply to private and federal public institutions. These
account for 90% of the over 2 400 HEIs in Brazil and enrol 91% of undergraduate students
in the country. Three quarters of undergraduate enrolment in Brazil is in the private sector.
The remaining 9% of enrolment is in state and municipal public institutions, which are
subject to regulation and quality assurance by state governments. CAPES evaluation
applies to all academic postgraduate education in the country.

This review assesses the relevance, effectiveness and efficiency of the quality assurance
procedures in place. The OECD review team was asked to consider the effectiveness and
efficiency of the systems in ensuring minimum quality standards, providing differentiated
measurement of quality and promoting improvement of quality and quality-oriented
practices in HEIs, providing recommendations for improvement.

Regulating “market entry”: ensuring providers of higher education meet high
standards, while streamlining quality assurance procedures

In contrast to some other countries in the Latin America and Caribbean region, compliance
with Brazil’s system of accreditation for private higher education institutions and official
recognition of undergraduate programmes is nearly universal. The requirements of
institutional accreditation are sufficiently rigorous to limit fraudulent or grossly unqualified
private institutions from entering the higher education marketplace. The formal
requirement for all new undergraduate programmes to obtain official recognition in the
early stages of their operation provides a basic guarantee of quality.

Nevertheless, Brazilian authorities could improve the relevance and effectiveness of the
regulatory and evaluation processes that govern the “market entry” of new private higher
education providers and undergraduate programmes. First, current quality assurance
systems and frameworks can be adapted to make them more effective. In the short-term,
measures should include creating a user-friendly online platform to provide students and
families with reliable information on the accreditation status and quality of higher
education programmes; developing more sophisticated indicators to assess the quality of
distance education and monitor its expansion; improving the selection and preparation of

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

14 | EXECUTIVE SUMMARY

peer-review commissions; and taking greater account of pedagogical processes and initial
results in on-site reviews of recently created programmes.

Second, there is scope to target the finite resources available for external evaluation of
higher education on institutions and programmes that present the greatest quality risks for
students and society. HEIs with demonstrated capacity in internal quality assurance could
be permitted to “self-accredit” their own programmes, following rigorous institutional
reviews. For institutions that remain subject to programme-level reviews, Brazilian
authorities could consider allocating certain tasks to a professional inspectorate, allowing
academic peer reviewers to focus on evaluating core aspects of the learning process.

Ongoing quality assurance of undergraduate programmes: improving measurement
of quality and better targeting of resources

Each year, students graduating from undergraduate programmes registered in particular
disciplines take a mandatory competency assessment: the National Examination of Student
Performance (ENADE). The formal objective of ENADE is to assess students’ acquisition
of knowledge and skills specified in National Curriculum Guidelines (DCN) and their
understanding of unspecified broader societal themes. These objectives are too broad for a
test with the scope of ENADE and the general knowledge component of the tests is
unrelated to the content of the programmes it is supposed to evaluate.

There are also weaknesses in the way ENADE is designed and implemented, which hinder
its ability to generate reliable information on student performance and programme quality.
It is a low stakes exam, which reduces the motivation of students; test items are not
standardised, meaning tests are not of equivalent difficulty between years and subjects and
there are no explicit quality thresholds to indicate what good performance is. Results for
students on each programme are standardised to generate a score on a scale of one to five,
but this is a relative measure of average student performance, not a clear indication of the
level of their knowledge and skills. Brazilian authorities should undertake a thorough
assessment of the objectives, costs and benefits of large-scale student testing to identify
how weaknesses can be addressed and how, in contrast to the current situation, ENADE
could be made into a useful tool and feedback mechanism for teachers and institutions.

The results of ENADE feed into a composite indicator of quality for each programme: the
Preliminary Course Score (CPC). This also includes scores for the profile of the teaching
staff, student feedback and an indicator of assumed learning gain (IDD). The IDD is based
on a number of bold assumptions about the influence of programmes on student
performance, which make it hard to justify its weight (35%) in the CPC. INEP should move
to monitoring programme performance using an “indicator dashboard”, with a broader
range of disaggregated indicators, including measures of student drop-out and, ultimately,
graduate employment outcomes.

Site visits for established undergraduate programmes, which are currently used only when
programmes perform poorly in relation to the CPC, use a review template and scoring
system that do not focus on identifying the causes of poor observed performance and do
not consider graduation rates and graduate destinations. Site visits should be retargeted to
focus on the root causes of poor performance highlighted by indicators, while peer
reviewers should also visit good programmes to help them understand the factors that affect
good performance.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

EXECUTIVE SUMMARY | 15

Quality in postgraduate programmes: fine-tuning existing practice and planning for
the future

The system of external quality assurance for academic postgraduate education in Brazil
sets a comparatively high bar for academic postgraduate training to enter the system.
However, there is scope for the peer-review committees that evaluate new programme
proposals to focus more on the relevance of programmes to expanding knowledge fields
and on the design of the training provided to postgraduate students. The four-yearly
periodic reviews involve resource-intensive review of staff outputs, alongside
consideration of other factors, but neglect training conditions, student output and graduate
destinations. CAPES should rebalance the evaluation criteria to focus more on student
outputs and outcomes. The reliance on peer review will make the system harder to scale as
postgraduate education expands, while inbreeding creates risks for objectivity and quality.
It is important to involve international peers in assessment of the top-rated programmes
and plan for the future, notably though evaluating the role of academic master’s
programmes and the real costs of the current peer-review processes.

Quality higher education institutions: increasing focus on internal quality assurance
and the role of external institutional reviews

Although private and federal public institutions are subject to periodic re-accreditation,
through institutional reviews, “de-accreditation” is rare for private institutions and
effectively impossible for public institutions. The General Course Index (IGC) — another
composite indicator used by INEP - provides limited signals about institutional quality. On-
site re-accreditation reviews pay limited attention to evidence of institutional performance,
internal quality processes and their practical implementation. Brazilian authorities should
reduce the period of accreditation for universities and university centres (currently eight or
ten years) and, in more robust institutional reviews, increase the focus on outputs, outcomes
and internal quality assurance procedures. A greater focus on these issues would allow
Brazil to move to a system where institutions with demonstrably strong internal quality
assurance capacity and a proven record of delivering quality can accredit (authorise and
recognise) their own programmes.

Governance of the quality assurance system: ensuring greater transparency,
improved steering and engagement of sector organisations

The basic legitimacy of external quality assurance in the federal higher education system
is not questioned and INEP has developed significant experience and capacity in
evaluation. However, challenges in system governance include a potential conflict of
interest in MEC, which both steers and regulates federal public institutions. CONAES - the
National Commission for Evaluation of Higher Education - lacks resources and the capacity
to oversee the quality assurance system, while higher education intermediary organisations
have weak capacity in quality-related matters. Drawing inspiration from international
examples, Brazil should explore ways to establish an independent quality assurance agency
to take on roles currently in MEC and INEP, considering the option of combining a
professional inspectorate and academic peer review. CONAES should receive dedicated
resources and sector associations in higher education should be incentivised to promote
quality across the system. In cooperation with state governments and the higher education
sector, federal authorities should explore how a reformed external quality assurance system
could also apply to state and municipal institutions.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 17

1. Assessment and recommendations

This chapter presents an overview of the main findings and recommendations resulting
from the OECD review of the external quality assurance system in the federal higher
education system in Brazil. The analysis has assessed the relevance, effectiveness and
efficiency of the external quality assurance procedures applicable to undergraduate and
postgraduate programmes and higher education institutions (HEIs) in the federal higher
education system. The chapter summarises findings and recommendations in relation to
the different components of the external quality assurance systems: a) procedures to
regulate the “market entry” of new private HEIs and new undergraduate programmes; b)
procedures for the ongoing monitoring and evaluation of existing undergraduate
programmes and related feedback and corrective measures; c) external quality procedures
governing academic postgraduate programmes; d) ongoing monitoring and evaluation of
higher education institutions (institutional evaluation) and; e) governance and
administrative bodies and arrangements that have been created to implement and oversee
the processes above.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

18 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

1.1. Focus of this chapter

A review of procedures for external quality assurance in the federal higher
education system in Brazil

This chapter presents an overview of the main findings and recommendations resulting
from the OECD review of the procedures for external quality assurance in the federal higher
education system in Brazil. In line with the terms of reference agreed with the Brazilian
authorities at the start of the project, the analysis has assessed the relevance, effectiveness
and efficiency of the external quality assurance procedures applicable to undergraduate and
postgraduate programmes and higher education institutions (HEIs) in the federal higher
education system.

A focus on relevance, effectiveness and efficiency

Specifically, the terms of reference ask the team to analyse the effectiveness and efficiency
of the different aspects of existing quality assurance systems in a) ensuring minimum
quality standards in educational provision; b) providing differentiated measurement of
quality (between types of provision and levels of quality offered) and; c) promoting
improvement of quality and quality-oriented practices in HEIs (quality enhancement). On
the basis of the analysis, the OECD review team was invited to provide recommendations
for improving the system.

An analysis structured around the main components of the system

In light of the organisation of external quality assurance in the federal higher education
system, the review has analysed the different functions of the existing quality system in
turn:

1. First, procedures in place to regulate the “market entry” of new private HEIs and
new undergraduate programmes.

2. Second, the procedures for the ongoing monitoring and evaluation of existing
undergraduate programmes and related feedback and corrective measures.

3. Third, the external quality procedures governing the “market entry” and periodic
evaluation of academic postgraduate programmes, coordinated by the Foundation
for the Coordination of Improvement of Higher Education Personnel (CAPES).

4. Fourth, the ongoing monitoring and evaluation of higher education institutions
(institutional evaluation) and related feedback and corrective measures.

5. Finally, the governance and administrative bodies and arrangements that have been
created to implement and oversee the above processes.

1.2. External quality assurance in Brazil: evaluation, regulation and supervision

Established systems of external quality assurance for undergraduate
programmes, postgraduate provision and higher education institutions

Brazil has well-established systems in place at national level to regulate the operation of
public and private higher education providers and assess and monitor the quality of their
teaching and learning activities. There are distinct external quality assurance systems for
the undergraduate (ISCED 6) and postgraduate (ISCED 7-8) levels of education. Both

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 19

systems involve the external evaluation of individual higher education programmes, while
the evaluation of higher education institutions (HEIs) is coupled with the processes for
quality review for undergraduate provision.

Most higher education in Brazil falls under the regulatory responsibility of the
federal government and a large proportion of enrolment is in the private sector

The external quality processes for HEIs and undergraduate programmes apply to higher
education providers that are legally considered to be part of the “federal higher education
system”. This comprises federal public HEIs and all private HEIs in the country. Of the
roughly 2 400 HEIs in Brazil, 92% (federal public and private) fall under the regulatory
responsibility of the federal government and these institutions, together, account for 91%
of undergraduate enrolment. 75% of total undergraduate enrolment in Brazil is in the
private sector. The remaining 9% of undergraduate enrolment is in state and municipal
public institutions, which are subject to regulation by state governments, not the federal
authorities. The system of quality assurance for academic postgraduate programmes,
implemented by CAPES, applies to all higher education providers in the country, including
state and municipal institutions.

The quality assurance system for higher education institutions and
undergraduate programmes involves regulation, evaluation and supervision

For HEIs and undergraduate programmes, the current system of quality assurance was
established by legislation adopted in 2004 (Presidéncia da Republica, 2004;1;). This
establishes a system based on the legal regulation of institutions and undergraduate
courses, currently undertaken by the Secretariat for Regulation and Supervision of Higher
Education (SERES) within the Ministry of Education (MEC). SERES grants official
accreditation to private HEIs and authorisation and recognition to individual undergraduate
programmes. It does so on the basis of the results of quality evaluations of institutions and
programmes, which form part of the National System of Higher Education Evaluation
(SINAES) and are coordinated by the Anisio Teixeira National Institute for Educational
Studies and Research (INEP), a semi-autonomous agency under the responsibility of MEC.
INEP uses a combination of on-site peer reviews, results from the National Examination of
Student Performance (ENADE) and programme-level quality indicators to evaluate HEIs
and undergraduate programmes. SERES also uses the results of ongoing evaluation and
monitoring of HEIs and programmes by INEP to inform its supervision of the federal higher
education system and, in some cases, impose remedial measures and sanctions on HEIs that
perform poorly.

CAPES evaluates academic postgraduate provision through a system of peer
review, the results of which have far-reaching consequences

The evaluation of academic postgraduate programmes (master’s, Professional Master’s and
doctoral programmes), undertaken by CAPES, involves an elaborate system of peer review,
organised by 49 discipline-specific field committees. The field committees approve all new
academic postgraduate programmes and undertake evaluations of existing programmes on
a four-year cycle. The results of the CAPES evaluations are fundamental for the approval
or new academic postgraduate programmes, the continued operation of existing
programmes and the allocation of public money for researcher training. Programmes which
fail to achieve minimum standards in CAPES evaluations lose funding and the official
validity of their qualifications, and are forced to close.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

20 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

1.3. Regulating market entry: new institutions and undergraduate programmes
Main findings

A system of institutional accreditation and programme-level recognition is used to
regulate market entry for HEIs and undergraduate programmes, mostly affecting
the private sector

Private higher education institutions are required to obtain formal external institutional
accreditation (credenciamento) from MEC to allow them to begin operation and may only
initially be established as teaching institutions, with the formal status of “college”
(faculdade). Once established, private institutions may transition to the status of “university
centre” (larger teaching institutions) or a fully-fledged “university” (institutions with
teaching and research activity). This is possible if the institutions meet certain criteria
related to number of programmes; the qualification and employment status of staff; and for
universities, research; and if they successfully complete a process of re-accreditation. In
2017, MEC processed over 200 requests for institutional accreditation from private
institutions.

Public HEIs may be established with any institutional form and are accredited
automatically by their acts of establishment. The establishment of new federal public
institutions is rare, but, from a legal perspective, such institutions can be created without
any requirement to undergo an initial external evaluation.

As a general rule, colleges in the federal higher education system, which are almost
exclusively private, are required to obtain formal authorisation from SERES (autorizagdo)
to start new undergraduate programmes. As discussed below, colleges with adequate
institutional quality ratings may be exempted from aspects of the authorisation process in
certain circumstances. Institutions with the status of university centre and university have
a greater degree of autonomy and are not generally required to obtain authorisation in
advance to start new undergraduate programmes, but must notify SERES of the creation of
all new programmes. Universities and university centres do require prior authorisation to
start new programmes in medical fields and law. In 2017, MEC processed nearly 1 600
requests for authorisation for new classroom-based and distance undergraduate
programmes.

All HEIs in the federal system, whatever their legal form, are required to submit new
programmes to an external quality assurance process called “recognition”
(reconhecimento), once half of total teaching hours for the first student cohort have been
completed (in the second or third year, for example). All new programmes offered by HEIs
need to complete the recognition process successfully for the degrees they award to be valid
nationally in Brazil. In practice, most of the programmes undergoing initial recognition are
also in new or expanding private sector institutions. Federal public universities, as well-
established institutions, already active in a broad range of study fields, create fewer new
programmes and are thus less involved in processes of programme recognition.

Decisions about accreditation, authorisation and recognition are informed by the
results of on-site peer evaluations coordinated by INEP

SERES makes decisions regarding the accreditation of institutions, and authorisation and
recognition of programmes, based on the results of on-site peer evaluations of the
institutions and programmes in question. Two or three external evaluators undertake review

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 21

visits, using evaluation criteria and scoring detailed in evaluation templates (“instruments’’)
specific to each process.

For the accreditation process, which applies only to new private institutions, the evaluation
template is organised around five thematic axes, and assesses the proposed institution
against 45 qualitative indicators, each of which is evaluated on a five-point scale. The
principal foci are the Institutional Development Plan (30%), planned academic policies
(20%), planned management policies (20%) and infrastructure (20%). The final score
generated by this evaluation, on a scale of one to five, is referred to as the “institutional
score” or Conceito Institucional (Cl). Institutions need to score at least three to receive
accreditation from SERES. Institutional accreditation for colleges is valid for three to five
years, depending on the CI score they receive. After this period, colleges must formally
undergo a process of re-accreditation (see discussion of institutional re-accreditation
below).

For the processes of programme-level authorisation and recognition, the on-site evaluation
instruments used by external reviewers establish nearly identical review templates. The
evaluation template for authorisation focuses on planned inputs (teaching staff,
infrastructure, etc.) and activities (pedagogical processes, support to students etc.) linked
to the programme. There are around 50 qualitative indicators in the template. These are
assessed by the external evaluation team appointed by INEP taking into account
programme documents, discussions with proposed staff and visits to the facilities planned
for the programme. The criteria in the template for recognition focus on the real inputs and
activities involved in the new programme, once the first cohort of students has completed
half of their study hours.

SERES can impose sanctions on HEIs if recently established programmes receive
negative evaluation results in the process of recognition

If the result of the on-site evaluation at the stage of recognition is negative (a score of two
or less), SERES requires the HEI to draw up a “Commitment Protocol”, which sets out how
the quality problems detected will be addressed within a 12-month timeframe. If it
considers there is an immediate risk for students, SERES can legally impose one or more
sanctions on the HEI providing the programme, including suspension of the right to recruit
new students. This rarely happens in practice. At the end of the period established by the
Commitment Protocol, the programme is subject to another on-site inspection by INEP
evaluators. If it still fails to meet minimum quality requirements, SERES can launch a
“sanctioning procedure”, which may entail the same range of sanctions. For serious cases
in private institutions, the relevant legislation allows for the withdrawal of institutional
accreditation, which would effectively lead to the closure of the institution. Again, in
practice, such cases are rare. Legally, some of the sanctions provided for in the legislation
can be applied to public institutions, but the legal status of these institutions as public bodies
means they may not have their institutional accreditation withdrawn.

Recent changes have removed the requirement for colleges to undergo a site visit
to obtain authorisation to start new programmes in specific cases

The authorisation of new programmes proposed by colleges is a risk-adjusted process.
Recent changes to the regulatory regime allow colleges to obtain authorisation for new
courses under certain circumstances, without undergoing an on-site inspection. Colleges
with the minimum institutional quality score (CI) of three can start up to three new
programmes a year without on-site reviews, provided they already have officially

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

22 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

recognised (i.e. quality assured) programmes in the same disciplinary field. Colleges with
institutional quality scores of four and five can create more new programmes a year in
fields where they already operate. In these cases, HEIs must still request authorisation, but
the procedure is based exclusively on a desk-based analysis by SERES of the programme
documents submitted by the HEI.

In contrast to some other countries in the Latin America and Caribbean region, compliance
with Brazil’s system of institutional accreditation appears to be nearly universal. Private
institutions do not frequently operate without institutional accreditation. The requirements
of institutional accreditation appear to be sufficiently rigorous to limit fraudulent or grossly
unqualified private institutions from entering the higher education marketplace. Moreover,
accreditation requirements do not appear to have created excessive barriers to the market
entry of private higher education providers. Brazil’s higher education system has grown
swiftly over the last decade, and private sector institutions have provided the majority of
new study places.

Some cases of fraud do exist, while information on the accreditation status of
institutions is not as accessible and transparent as it could be

Nonetheless, there are examples of accredited higher education institutions offering
programmes that are not authorised, and organisations that are not accredited higher
education institutions offering fraudulent diplomas. While the Ministry’s e-MEC platform
provides a single national registry of accredited institutions and authorised programmes, it
is primarily an administrative database. Incidents of allegedly fraudulent provision suggest
that not all students have ready access to information that allows them to confirm the
validity of the institutions and programmes in which they plan to study. While the layout
and functionality of the e-MEC site are not designed to be used by students and their
families, the information contained in the system could easily be exploited as part of a more
user-friendly information service.

Despite some strengths, there are concerns about the rules governing distance
education providers and programmes

Distance education now accounts for almost 20% of total enrolment in Brazil, with over
90% provided by the private sector. Private distance education institutions and the
programmes they provide are subject to the same procedures for institutional accreditation
and programme-level authorisation and recognition as providers of traditional classroom-
based higher education. A limited number of qualitative indicators relating specifically to
distance education have been incorporated into the evaluation templates used for
accreditation, authorisation and recognition, covering pedagogical approaches, digital
technologies and infrastructure. For example, evaluators are called on to consider the
capacity of teaching staff and assistant tutors to support and mentor the number of students
proposed for each programme (the proposed study places). Brazilian legislation requires
distance education programmes to respect the requirements of national curriculum
guidelines (DCNs), for fields where these exist, and distance programmes have hitherto
mostly been blended programmes, with some face-to-face instruction and assessments,
often conducted in decentralised distance education learning centres (referred to as

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 23

“poles”). Internationally, blended programmes have been shown to be more effective than
fully online programmes (Escueta et al., 2017,2)).

Recent legislative changes have made it easier for private higher education providers to
establish large numbers of distance education “poles” (up to 250 a year), in multiple
locations, without the need for the facilities in each location to be inspected by INEP
evaluators. Some stakeholders in Brazil are concerned that this will promote the
uncontrolled expansion of distance education, without adequate quality guarantees.

The system for programme-level authorisation and recognition creates additional
guarantees of minimum quality standards

The formal requirement for all courses to obtain official recognition in the early stages of
their operation provides a basic guarantee of the quality of programmes. The procedures in
place force higher education providers to reflect seriously about the design of the
programmes they are providing and put in place a range of policies and processes —
described in the Programme Pedagogical Project (PPC) — that should contribute positively
to the delivery of relevant programmes meeting minimum quality criteria. Nevertheless,
the factors verified through the on-site evaluation at the stage of recognition are all
conditions for the delivery of quality programmes, rather than indicators of the initial
performance of the programmes in question (student progression and performance, for
example). Moreover, the processes used to evaluate the quality of new and recently created
programmes are subject to various lines of criticism.

There have been concerns about the profile and objectivity of the review
commissions undertaking on-site reviews

Representatives of private institutions consulted by the OECD review team complained
that the composition of the reviewer pool used to implement on-site reviews is often skewed
towards public universities, while institutional representatives more generally argued that
those who are called upon to carry out reviews sometimes lack expertise concerning the
programme under review. There is also concern about the subjectivity or unreliability of
qualitative assessments. The revised process of on-site review for programme authorisation
and recognition, as amended in late 2017, asks reviewers to make qualitative judgements
on a five-point Likert scale, using pre-formulated judgement criteria. Despite the attempts
by INEP to formulate the judgement criteria clearly, these scales still leave considerable
room for interpretation. They call upon reviewers to make distinctions that are likely to be
inconsistent between individuals. The OECD review team was told by campus officials that
the same programme offered in different campuses with otherwise near-identical supply
conditions received different marks from on-site reviewers.

The set of indicators used in the evaluation templates and the weight accorded to
different topics are not optimal

The on-site evaluation instrument used by external reviewers for the process of programme
recognition assigns 40% of its weight to assessment of the teaching staff attached to the
programme. This reflects the fact that the staff are working at the time of recognition, so

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

24 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

the composition of the teaching workforce can be judged more accurately than during the
previous process of authorisation, where this is used. However, the judgement criteria
reward the presence of full-time staff with doctoral degrees and attach little value to
professional experience, thus disadvantaging professionally oriented programmes. At the
same time, relatively little weight is attached to assessment of the pedagogical and didactic
approaches implemented by the programme, despite their crucial role in supporting
students to acquire relevant learning outcomes.

The on-site evaluation templates now make special provision for the authorisation and
recognition of distance education courses. However, 45 out of 55 indicators in the templates
apply to both classroom-based and distance programmes. The specific indicators of
programme quality related to curriculum, instruction, learning support, and assessment in
distance programmes are less developed than those used in accreditation systems in other
OECD and partner countries, including the United States. Developing appropriate
measures of quality that reflect the specific characteristics of distance education is,
however, a challenge shared by many higher education systems.

Finally, on-site visits carried out for programme recognition permit higher education
institutions to award degrees without providing evidence about the initial performance of
the programme, such as rates of attrition among its students in the first years of operation.
Additionally, the process of recognition does not systematically elicit information from the
students whom the programmes serve, or external stakeholders who have experience of
working with the programme and its students, such as public sector employers and private
firms which provide internships.

The processes of programme-level authorisation and recognition are
administratively burdensome for HEIs and INEP

The OECD review team heard frequent criticisms from institutional representatives of the
delay and burden associated with the on-site review process for authorisation and
recognition. INEP and SERES argue that the situation has improved in the last two years.
In particular, they point to the fact that HEIs that have received adequate quality scores (a
CI of three or above) are exempted from on-site reviews at the stage of authorisation for
programmes in fields where they already have courses (within certain limits). They argue
that the most recent regulatory changes in Decree 9 235/2017 reduce burden for institutions
with an established quality record, allowing them to create additional study places more
easily, for example.

This contrasts with the situation in many OECD and partner countries, where HEIs can
create new programmes and issue valid diplomas without prior programme-level
authorisation. In these and other systems, authorities also often link quality review
procedures more closely to risk of poor quality than is the case in the Brazilian system,
with less complex procedures in place for institutions that can demonstrate they present a
lower quality risk. Although the large private higher education sector in Brazil creates
specific risks, which are not found in all higher education systems, there is certainly scope
for Brazil to draw on risk management practice in other quality assurance systems.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 25

Key recommendations

Although MEC, with the support of evaluations coordinated by INEP, regulates the entry
of new institutions into the Brazilian higher education marketplace more comprehensively
than in other systems undergoing rapid expansion, the quality assurance system is not fully
effective in preventing fraudulent and unauthorised provision. The first line of defence
against unaccredited higher education providers is students themselves. Informed students
understand which institutions are accredited and not, and why this matters to them, and are
able to identify and avoid unaccredited institutions. In principle, comprehensive
information about accredited institutions and recognised programmes is available through
the online e-MEC. However, e-MEC is not a user-friendly source of accreditation
information. More accessible public Internet resources found in other higher education
systems could serve as references for the Brazilian authorities in this regard. In the medium-
term, the aim should be to develop a comprehensive online portal providing students and
prospective students not only with programme-level information on quality assurance
results, but also on issues such as graduation rates and graduate employment outcomes (see
discussion on programme indicators below).

Despite attempts to address concerns about the composition of review commissions and
reduce requirements for authorisation in some cases, the Brazilian system of programme
review at market entry remains complex and burdensome and may not represent the best
use of the country’s resources. Programme-focused regulatory decisions — for new and
existing programmes - account for more than 10 000 of the 12 000 acts that SERES handles
annually. The Brazilian system of quality assurance currently focuses proportionally more
efforts on the programme-level than on the institutional level as a unit of evaluation and
monitoring.

‘self-accrediting status. Quality guarantees could be maintained across the system by an
enhanced system of programme-level monitoring indicators and more rigorous and

comprehensive process of institutional re-accreditation.

3. In the near term, take steps to improve the evaluation process for programmes
that remain subject to programme-level authorisation and recognition

The OECD review team sees a clear case for maintaining programme-level authorisation
record of good quality provision and are not able to demonstrate adequate capacity to self-

accredit their own programmes. It is thus important to increase the effectiveness of these
processes in promoting quality practices for institutions that remain subject to programme-
level authorisation and/or recognition. Priorities for improving current practice in the short-
term include:

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

26 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

e Further improving the criteria used to select and assign peer reviewers for on-site
reviews to increase the fit between reviewer expertise and programme review
responsibilities.

e Continuing and increasing efforts to improve the training of peer reviewers, with a
view to improving the reliability and impartiality of scoring.

e Increasing the weight attached to the organisation and implementation of teaching
and learning in the evaluation instrument for recognition, reflecting the importance
of these factors for students.

e Using the recently introduced process of feedback about the performance of peer
reviewers to monitor and revise selection and training.

4. In the longer term, take steps to reduce further the burden and improve the
effectiveness of quality assurance processes for programmes outside self-
accrediting institutions

In the longer term, two issues should be considered in particular. First, the procedures for
on-site visits could be fundamentally reformed. Responsibility for reviewing institutional
infrastructure and basic institutional policies could be assigned to a well-trained and
‘professionalised inspectorate. The expert judgement of academic peers (who currently
review all aspects of institutions and programmes) could then be applied to a more limited
set of indicators than at present, focused on core teaching and learning activities. A
sequenced process of accreditation and authorisation could be implemented in which a
professional inspectorate initially carried out its work, and academic peers would be
engaged only for institutions and programmes that have passed a first stage of review.

‘Second, it will be important to identify ways in which the more extensive, quantitative, and
‘comparable information about intermediate programme performance can be incorporated
into the process of programme recognition, Examples include student attrition from

programmes, and student feedback concerning the teaching and learning environment.
1.4. Assuring and promoting quality for existing undergraduate programmes
Main findings

Student testing, programme indicators and on-site reviews all play a part in the
ongoing quality assurance of undergraduate programmes

Once undergraduate programmes have been recognised, they are subject to an ongoing
cycle of evaluation coordinated by INEP on behalf of the Ministry of Education. As
currently designed, this cycle involves the collection and collation, by INEP, of
programme-level data, including the results of the programme’s students in a national
assessment of learning outcomes, to create a composite indicator of programme
performance every three years. As a general rule, in cases where programmes score low
ratings in relation to this composite indicator, a new on-site programme review is
undertaken by external evaluators. The results of a programme in relation to the composite
indicator and, where used, the on-site review determine whether or not SERES renews its

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 27

official recognition, thus guaranteeing that the diplomas awarded by the programme retain
national validity.

ENADE is a set of tests used to measure the performance of students in
undergraduate programmes

Each year, students graduating from undergraduate programmes registered in a particular
set of disciplinary fields are required to take a mandatory competency assessment — the
National Examination of Student Performance (ENADE). Disciplines are assigned to three
broad groups, with disciplines in group I evaluated one year, group II the year after and
group III the year after that, meaning each discipline is subject to ENADE every three
years. The ENADE tests contain a general competency assessment, common to exams in
all fields in a single year, and a discipline-specific component. There are currently separate
ENADE tests for 88 study fields, with discipline-specific questions for each test developed
by multiple academics selected by INEP. In addition, all students participating in ENADE
are required to complete a student feedback questionnaire providing biographical
information and a personal assessment of their programme. The average results obtained
by students in each programme are used to calculate an “ENADE score” on a five-point
scale for that programme. These scores are published and also feed into the composite
indicator of programme quality discussed below.

The objectives established for ENADE by the legislator are unrealistic

The objectives of ENADE, as currently formulated, are unrealistic. The legislation
establishing SINAES requires ENADE to measure students’ performance in relation to the
content of relevant national curriculum guidelines, their ability to analyse new information
and their wider understanding of themes outside the scope of their programme. The
requirement to measure understanding of unspecified “themes outside the scope of the
programme” - which has given rise to the general competency assessment in ENADE - is
inherently problematic because it is so general and the knowledge and skills assessed, by
definition, are not part of the programme’s core intended learning outcomes. It is thus
unclear how those running programmes could be expected to equip students with such a
range of unspecified knowledge and skills or why they should be held accountable for
students’ not having these competencies at the end of their studies.

As ENADE is a written examination with a restricted duration, it is also impossible to
measure the full range of learning outcomes that any adequately formulated curriculum
guidelines should contain. Moreover, by implying that ENADE sets out to measure
students’ learning outcomes in relation to the National Curriculum Guidelines for
undergraduate programmes, there is a risk that the content of ENADE (in a given year or
over several years) comes to be seen to define what is important in the National Curriculum
Guidelines. If ENADE is to be maintained, Brazil’s legislators and quality assurance
authorities need to provide a more credible account of what it can realistically achieve and
how risks for innovation and responsiveness can be mitigated.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

28 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

There are significant weaknesses in the way ENADE is currently designed and
implemented, which undermine its ability to generate reliable information on
student performance and programme quality

The OECD review team considers that there are at least five principal weaknesses in the
way ENADE is currently designed and implemented:

1. The first problem relates to the participation of students and their motivation to
make an effort in the test. A proportion of the students who should be taking the
test each year are not doing so. Across years, between 10-15% of students registered
to take the test each year do not turn up on the day. Moreover, there are concerns
among stakeholders in Brazil that some HEIs seek to avoid registering a proportion
of students for ENADE. At the same time, ENADE is a high stakes exam for HEIs
(as it is used in the quality rating of their programmes), but a low stakes exam for
students. Although attendance is compulsory, ENADE scores have no effect on
students’ academic record and there is evidence that a significant proportion of
students do not complete large parts of the test. Evidence from other OECD and
partner countries suggests that if the results of tests have no real consequences for
students, this impacts negatively on student motivation and performance. Low
student motivation is likely to have negative implications for the validity of
ENADE results as an accurate reflection of the learning outcomes of students.

2. A second concern relates to the development, selection and use of test items for
each ENADE test. At present, there is no robust methodology to ensure that the
difficulty of each test item is taken into account in the composition of the test and
thus that a) tests in the same field are of equivalent difficulty between ENADE
cycles and b) tests in different fields are of a broadly similar level of complexity.
This means that it is not possible to compare the raw results or the Conceito ENADE
between years or between disciplines. A related question is whether the number of
discipline-specific items included in ENADE (30) is adequate to generate a reliable
indication of students’ learning outcomes from an undergraduate programme. The
answer almost certainly depends on what the exam is for. A robustly designed
examination with only 30 items may be able to provide a general indication of a
students’ level of knowledge and competencies in a specific disciplinary field.
However, such a test is unlikely to provide reliable evidence of students’
performance in specific sub-fields or aspects of the curriculum, which limits its
usefulness as a tool to help teaching staff and institutions improve the design of
their programmes.

3. A third problem is that no explicit quality thresholds or expected minimum levels
of performance are set for ENADE tests. Without tests of a comparable standard of
difficulty and without defined quality thresholds (pass, good, excellent, etc.),
ENADE scores are simply numbers. It is impossible to know if students in
programmes that achieve 50% or 60% in ENADE are performing well or poorly.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 29

5. A final issue is that the standardisation of ENADE scores compounds the lack of
transparency about what ENADE results really mean. Raw marks are attributed to
a five-point scale based on the standard distribution of scores in a single subject in
a given year. As tests may vary in difficulty and students obtain very different
distributions of scores, where a programme falls on a standard distribution of the
scores for all programmes says little about the actual quality of the programme in
question.

These reliability issues and the limited use of ENADE results for quality
improvement by HEIs call into question the resources dedicated to the exam

The results of ENADE are used by INEP and SERES for regulatory purposes, as discussed
below. However, institutions consulted by the OECD review team report that they did not
use of ENADE results in efforts to improve the design and content of programmes.
Representatives of institutions consistently indicated that they did not see ENADE as
providing useful feedback to help them improve their programmes. Although the OECD
review team does not have access to a detailed breakdown of the costs of implementing
ENADE, these account for a substantial part of INEP’s budget for evaluation of higher
education, which amounted to over 118 million reals (USD 30.7 million) in 2017. It is
questionable whether the quality and usefulness of the results achieved with the exam as
currently configured justify the investment of public resources committed.

The Preliminary Course Score (CPC) is used as a composite indicator of
programme quality

To monitor programme performance, INEP currently uses a set of indicators comprising a)
measures of student performance and assumed learning gain (based on ENADE test
results); b) the profile of the teaching staff associated with the programme and; c) feedback
from students about teaching and learning, infrastructure and other factors from the
questionnaires they complete in advance of taking the ENADE test. When new ENADE
results are available for each programme, after each three-year cycle of testing, INEP
calculates a programme score — the Preliminary Course Score (CPC). Programmes that
score below three out of five on the CPC are systematically subject to on-site inspections
by external review commissions, with a positive evaluation score (a CC of three or above)
a prerequisite for renewal of their official recognition. Courses that score three or above on
the CPC generally have their programme recognition renewed automatically by SERES,
without having to undergo an on-site inspection.

The CPC is an unreliable measure of quality, lacking transparency

The CPC attributes 35% of its total weight to an “Indicator of difference between observed
and expected performance” or IDD. This is calculated by comparing each student’s actual
results in ENADE with the performance that would be expected given their previous
performance in the national high-school leaving exam, ENEM. The combination of the
boldness of the underlying assumptions about the predictive value of ENEM results for the
performance of undergraduate students; the poor reliability of ENADE results in the first
place; and the potential influence of factors outside the control of the programme on student
performance mean that the IDD provides only limited information on programme quality.

Moreover, it is widely accepted that the weightings attributed to the different indicators in
the CPC are arbitrary, with no discernible scientific basis (TCU, 2018,3)). This further
compounds the lack of transparency about what the scores attributed to courses really mean

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

30 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

in practice for students, families and society at large. It is positive that the CPC sets out to
include indicators of the teaching process (through the imperfect proxy of teaching staff
status); qualitative feedback from students (the main beneficiaries of the system) and
measures of student learning outcomes. It does not, however, contain a measure of the
attrition rate of students (what proportion of students entering a programme complete it) or
the subsequent employment outcomes of students.

The principle of using indicators to identify “at risk” programmes and target finite
resources for on-site inspections makes sense, especially in a system as large as Brazil’s.
However, the CPC does not provide a reliable mechanism to identify poorly performing
courses. The absence of quality thresholds in ENADE and the standardisation processes
used to create the ENADE score, combined with the weaknesses of the IDD, mean it is far
from clear whether a CPC score of three represents an adequate standard of quality or not.
A reform of the monitoring indicators used and the way they are combined is necessary.

Site visits are undertaken for programmes that perform poorly on the CPC

When programmes are identified through the CPC as performing poorly — often meaning
they have poor relative performance in ENADE — they are subject to an on-site inspection
by external evaluators, coordinated by INEP. The evaluators assess the supply conditions
for the programme using the same evaluation instrument that was already used for
programme recognition (reconhecimento). The results of the new on-site inspections are
used by SERES as a basis for decisions for the renewal programmes’ official recognition.
The evaluation attributes a new quality score — an updated Conceito de Curso (CC) — that
effectively replaces the CC attributed at the time of initial recognition and exists alongside
the CPC score in the e-MEC system.

These site visits use a review template and scoring system that do not focus on
identifying the causes of poor performance in the CPC and do not consider
graduation rates and graduate destinations

The on-site visits for renewal of recognition, as currently organised, (re)check compliance
with basic standards that was already checked through the initial on-site visit for the
recognition of the programme. The evaluation instrument for recognition and renewal of
recognition places a 40% weighting on the category “teaching staff’ and 30% on
“infrastructure”, with just 30% attributed to teaching and learning policies and practices.
The indicators and judgement criteria relating to teaching staff mostly focus on the
qualifications and experience of the individuals in question, with only three indicators
dealing with the activities (atuacdao) of staff or their interaction with each other.

As such, the renewal of recognition on-site reviews do not focus strongly on the teaching
and learning-related factors that might be expected to have greatest influence on student
performance and quality. Frequently, it appears that programmes which score poorly on the
CPC measure subsequently achieve a higher score on the CC (TCU, 2018)3]). As such, these
programmes nominally recover the higher quality score. It is understandable that the CPC
and the inspections leading to the CC can generate different values, as they measure almost
entirely different things. A greater focus on teaching activities and the greater attention to
outputs and outcomes (attrition rates, learning outcomes, graduation rates and employment
outcomes) would make the evaluation instrument more effective in identifying the real
causes of poor performance.

More generally, the objective of targeting on-site inspections on weakly performing
programmes has advantages, as the systematic use of periodic on-site inspections for all

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 31

existing programmes in Brazil would almost certainly be unfeasible for logistical and
financial reasons. However, it also means programme-level site visits at this stage in the
evaluative process always have a punitive character and that peer reviewers are not exposed
to good practice in well-established programmes, which could inform their judgements
about, and recommendations to, poorly performing programmes.

Key recommendations

1. Undertake a thorough assessment of the objectives, costs and benefits of large-
scale student testing as part of the quality assurance system

Officially ENADE currently seeks to assess students’ acquisition of knowledge and skills
specified in the relevant National Curriculum Guidelines (DCN) or the equivalent
documents for Advanced Technology Programmes, as well as their understanding of
unspecified “themes outside the specific scope” of their programme. This is an unrealistic
objective and no standardised test could achieve this. Moreover, as discussed in the
preceding analysis, the current design and implementation of the ENADE tests are
characterised by significant weaknesses. At present, ENADE results are used extensively
as a basis for regulatory decisions (renewal of programme recognition), but are not used by
institutions and teachers to identify areas where their programmes need to be strengthened.

The OECD team believes that, in its current form, ENADE does not represent an effective
use of public resources. As such, as a basis for decisions on the future of the system, a
thorough reflection is needed about the objectives of large-scale student testing in Brazilian
higher education and the costs and benefits of different approaches to implementing it. The
main questions to answer are:

1. Can an improved version of ENADE, addressing the current design and
implementation weaknesses noted in this report, be implemented and generate
reliable information about the quality of undergraduate programmes?

2. Could the information about the quality of programmes generated by a revised
ENADE be provided by other, potentially more readily available, indicators? What
is the specific and unique added value of ENADE results?

3. Ifarevised version of ENADE does indeed have the potential to generate valuable
information that cannot be obtained from other sources, does the value of this
information justify the costs of implementing ENADE? How can the costs of
implementation be minimised, while still allowing ENADE to generate reliable and
useful results?

sampling
techniques could be deployed to reduce costs, while maintaining reliability.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

32 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

2. Ifa reformed version of ENADE is retained, ensure the objectives set for the
exam are more realistic

If the decision is taken to maintain a revised version of ENADE, it is crucial to ensure the
objectives set for it in the relevant legislation and implementing decisions are realistic and
clearly formulated. The objective of a reformed ENADE could be to provide:

e An indication — rather than a comprehensive picture - of the performance level of
students in relation to intended learning outcomes, as one indicator, alongside
others, in a comprehensive system of external quality evaluation and;

e Data on student performance that can be used directly by teachers and institutions
in identifying weaknesses in their programmes as a basis for improvement (quality
enhancement).

To achieve these objectives, the test should focus on measuring knowledge and skills that
programmes explicitly set out to develop in their students. This means abandoning claims
to measure abstract general knowledge with no direct link to the programme and focusing
on a) selected discipline-specific knowledge and skills and b) generic competencies that
can realistically be developed in an undergraduate programme. The latter category might
include critical thinking and problem-solving. These can theoretically be tested for using
discipline-specific test items.

3. Improve the design of ENADE tests to ensure they generate more reliable
information on learning outcomes that can also be used by teachers and HEIs

If maintained, ENADE tests should be designed in a more rigorous way to ensure that they
are of comparable levels of difficulty within subjects from one year to the next and that
tests for different disciplines are of equivalent difficulty for equivalent qualifications
(bachelor’s, Advanced Technology Programme, etc.). This may require a shift from classic
test theory to item response theory. As part of this process, performance thresholds and
grades should be established clearly in advance. The objective should be to provide students
and programmes with easily understood and usable grade point averages and grade
distributions. The approaches to both test design and performance thresholds used by
CENEVAL in Mexico or testing organisations in the United States might provide valuable
inspiration on how a revised form of the ENADE tests could be developed. It is important
for INEP to draw on the expertise of other organisations involved in standardised testing
internationally in the development of new approaches and test formats, to ensure it benefits
from a wide range of expertise.

4. Explore ways to make the results of ENADE matter for students

If maintained, ENADE needs to be made into a higher stakes exam for students, so that
they make an effort to demonstrate the level of knowledge and skills they possess.
Currently, it is difficult to make ENADE results count towards individuals’ degree scores,
not only because of institutional autonomy, but because only every third cohort has to take
ENADE. Including ENADE in degree results may be perceived as unfair to students in the
years where the test is applied. As a minimum, the ENADE score could be included in the
student’s diploma supplement. Alternatively, ENADE could be made into a curriculum
component for the years in which, or — in the case of sampling - for the students to whom,
it is administered, with the requirement that an equivalent test for students in other years
be administered by institutions. It is not yet clear if this would be possible legally.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 33

5. Introduce a new indicator dashboard, with a broader range of measures, to
monitor programme performance and identify “at risk” programmes

The use of the Preliminary Course Score (CPC) cannot be justified in its current form for
the reasons discussed above. However, systematic programme-level data are a crucial tool
for monitoring a system as diverse and variable in quality as Brazil’s. The most promising
option would be to include a broader set of more transparent indicators in an ongoing
monitoring system, with thresholds established to indicate “at risk” performances on
different indicators. This information could then be used to inform regulatory decisions and
feed into subsequent evaluation steps (such as on-site reviews). The system should apply
to all programmes, with data obtained from institutions and other sources, as appropriate,
and consolidated in a renewed version of e-MEC.

Such a system could use a more diverse set of indicators of teaching staff, real (not
standardised) ENADE results (based on established performance thresholds), an indicator
of drop-out rates and, when possible through linking data sources using the national identity
number (CPF), information on employment rates and earnings. Indicators of the socio-
economic profile of students could be included in the system, with higher tolerances for
issues like drop-out or ENADE performance for programmes with intakes from lower
socio-economic groups. Such variation in tolerances should be limited, as all students
should be expected to reach minimum standards and all programmes maintain a certain
proportion of their students. A revised form of the IDD could potentially be maintained
alongside the other indicators in the indicator dashboard, provided its status as a proxy for
expected performance and its limitations are made clear, and its weight in the overall
monitoring system is reduced.

The OECD review team understands that INEP is already planning (October 2018) to
“disaggregate” the components of the CPC and complement these with additional
indicators to inform the regulatory process. Hopefully, this recommendation will support
this process.

6. As part of a new system of institutional accreditation, exclude institutions with
demonstrated internal quality assurance capacity from on-site programme
reviews for the duration of their accreditation period

There is scope to exempt institutions from systematic ongoing programme-level review
that have a track record of good performance and that can demonstrate a high level of
internal quality assurance capacity. As discussed below, this would require existing
systems for institutional accreditation and re-accreditation to be strengthened. If problems
were identified through programme indicators in the indicator dashboard, in the first
instance, such institutions would be responsible for addressing these issues internally.
Addressing poor quality would become a key focus of institutional review and poor
performance or failure to address problems adequately could lead to institutions losing self-
accrediting status in the subsequent round of institutional review. This move would further
reduce some of the burden of external programme-level reviews for renewal of recognition
(as well as the initial recognition process).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

34 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

7. Maintain programme-level supervision for other institutions, with targeted on-
site reviews for poorly performing programmes and randomly selected highly
performing programmes.

For the remaining institutions, programme-level review would be maintained. The new
programme-level indicator dashboard (which would cover all programmes, including in
self-accrediting institutions) would allow poor programmes to be identified and replace the
current CPC system. If annually collected data on completion rates and employment
outcomes were included in the dashboard, alongside input indicators and periodic results
from a reformed ENADE, this would allow more effective continuous monitoring of
programmes. Problematic programmes could first be called upon to submit an improvement
plan that could be assessed remotely, largely in line with current supervision procedures.
SERES, or a future quality assurance agency (see below), could decide on timeframes for
improvement and whether and when an on-site visit would be required. It is crucial that
SERES, or a successor agency, have the capacity to close poor programmes rapidly if
programme indicators fail to improve without clear justification and evaluators give a
negative assessment following an on-site inspection.

However, while targeting of resources is important, the risk of evaluators only being
exposed to poor quality programmes — and thus lacking good reference points — needs to
be addressed. As such, it is recommended that reviewers also take part in reviews of
randomly selected programmes that obtain good scores in relation to monitoring indicators
- potentially including programmes in “self-accrediting” institutions - to allow them to gain
more insights into the range of practices and performance that exists in their field in the
country.

8. Develop a separate evaluation instrument for on-site reviews of established
programmes

The current process for on-site reviews of established undergraduate programmes uses the
same evaluation and judgement criteria as the instrument for programme recognition
(which occurs when the first student cohort has completed between half and three-quarters
of the programme). This instrument pays insufficient attention to programme outputs and
outcomes (notably the results of (a revised) ENADE, attrition and graduation rates and
employment outcomes) and to the teaching and student support practices that would be
expected to have the greatest influence on these outputs and outcomes. A new instrument
should thus be developed for on-site reviews of established programmes, which places most
emphasis on these factors. The earlier suggestion for an inspectorate to examine
infrastructure and basic institutional policies would mean that site visits by peer reviewers
could focus exclusively on the learning environment and possible causes of poor outputs
and outcomes.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 35

1.5. Assuring the quality of postgraduate education
Main findings

A dedicated system for the evaluation of academic postgraduate programmes

The system of external quality assurance for academic postgraduate education in Brazil
began in its current form in 1998. It evaluates and regulates academic (stricto sensu)
master’s programmes and doctorates. In Brazil, stricto sensu master’s courses - including
so-called “Professional Master’s” - are widely understood as the first stage in an academic
or research career - a situation that is largely a reflection of the relatively recent expansion
of doctoral education in the country. In many other OECD higher education systems,
master’s programmes, where they exist, are seen primarily as an extension and deepening
of undergraduate education, preparing graduates for a wide range of high-skill jobs in the
economy.

The CAPES evaluation system includes a specific approval process for new courses
(APCN), designed to ensure only academic teams with demonstrated expertise, a proven
track record of quality research and adequate facilities are authorised to provide academic
postgraduate education. Course proposals are assessed by a field committee composed of
academic peers from the field in which the course seeks to operate. Following a standard
assessment and validation process, new courses are formally approved if they score at least
three on a nominal scale of one to five. Every four years, CAPES implements a
comprehensive evaluation of all academic postgraduate programmes that have already been
accredited and been in operation sufficiently long for students to have produced academic
results. The results of this evaluation - attributed through scores on a scale of one to seven
- allow programmes to continue operating or, in case of poor performance, lead to
withdrawal of funds and recognition for the diplomas they award. This effectively means
programmes that fail the CAPES evaluation are forced to close.

The system for approval of new programmes sets a high bar for entry to the
system, but there is scope to review the balance of quality indicators used

The APCN process consciously sets a comparatively high bar for entry into the system of
academic postgraduate training and for the creation of doctoral training provision in
programmes that already operate at master’s level. In so doing, it seeks to maintain high
minimum standards for postgraduate education, protect students against poor quality
provision and ensure efficient targeting of public funding. During the review visits, the
OECD team noted a high degree of support for the principle of maintaining a high threshold
for entry into the academic postgraduate education system.

The criteria examined in the process for approval of new courses cover a wide range of the
variables that might reasonably be expected in an ex-ante assessment of a proposed
postgraduate programme. However, the current evaluation system pays comparatively
limited attention to the relevance of new courses to national or regional needs and
developing knowledge areas; to the design of the training programme; and to support and
personal development opportunities offered to students. Although the coherence of the
proposed course with the Institutional Development Plan (PDI) of the host institution is
assessed, there is no explicit assessment of the relevance of the course to the needs of
Brazil, in terms of knowledge development and highly qualified human resources.
Similarly, there is little obvious room in the evaluation templates to assess how the training
programme will help to develop students’ knowledge and skills and monitor their progress.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

36 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

The four-yearly periodic reviews involve resource-intensive review of staff
outputs, while neglecting training conditions, student output and graduate
destinations

Every four years, the field committees draw on information on staff, students, graduates
and details of scientific outputs reported by each postgraduate programme through the
online Sucupira platform, as a basis for their assessment of each programme. The quality
of student publications and the quality of the academic output of staff in academic journals
are assessed using a standard classification of publication “vehicles”, recorded in an online
database called Qualis. The assessment of books and book chapters is undertaken by
physically reviewing a sample of publications for each programme in depth, but represents
one of the largest calls of the time of members of some field committees (notably in the
humanities, social sciences and some of the hard sciences).

The set of indicators used in the CAPES four-yearly evaluations covers many of the key
variables that would widely be assumed to contribute to high-quality postgraduate
provision. It is positive that the evaluation grid, under different headings, takes into account
factors such as staff-to-student ratios, time to graduation and cooperation networks with
external research and non-academic organisations, for example.

However, the most striking feature of the four-yearly reviews is the strong focus on the
scientific output of the academic staff involved in the programmes being evaluated. The
CAPES evaluation is — nominally at least — an evaluation of postgraduate training
programmes, not a research performance evaluation. As such, it is questionable why the
system does not allocate less weight and fewer resources to assessing the performance of
staff and more to assessing the performance of students and outcomes of graduates.
Although there is also some attempt in the current CAPES system to assess the destinations
of graduates from programmes, this aspect of programme performance is not currently
addressed adequately.

The reliance on peer review will make the system harder to scale as postgraduate
education expands, while inbreeding creates risks for objectivity and quality

Despite the strengths of the current division of responsibilities within the CAPES
evaluation system, the evaluation system relies heavily on the voluntary contribution of
academic staff organised in discipline-specific field committees. Although academics
involved in the CAPES evaluation process consulted by the OECD review team felt the
time and effort required of them for the current system for approval of new programmes
remained reasonable, they highlighted that the CAPES system as a whole is becoming
unmanageable for field committees, as the number of postgraduate programmes increases.

The Review team understands that no assessment of the value of the time dedicated to
evaluation of courses by academic staff in the field committees — and thus also the cost to
their home institutions - is currently available. Given the comparatively rapid rate of
expansion of postgraduate provision in Brazil in recent years and the related increase in the
number of proposals for new courses, it will be important to develop a better understanding
of the number of person-hours used in the evaluation process and the associated costs.

The reliance on disciplinary committees composed exclusively of Brazilian academics also
risks creating an excessively narrow academic focus in evaluations. While scientific
excellence and traditional measures of academic output remain the basis for postgraduate
education, it is important to complement this with perspectives from outside academia, to
ensure that the development of postgraduate education responds to broader national and

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 37

regional needs. Moreover, the current process for the evaluation of new courses involves
limited or no direct interaction between those proposing the new courses and those
evaluating the proposals.

A second key issue with the staffing of CAPES evaluation processes is the risk of
endogamy (inbreeding). Even in a country size of Brazil — particularly given the relatively
small size of its postgraduate training system — the number of established academics in a
given field of study is limited. The number working in very high-quality departments and
programmes at an international level is even smaller. As such, there is the risk that the
people making judgements on whether or not a given programme is of international
standard have close connections with the programmes they are judging. Moreover, the
comparatively small pool of evaluators and their background may lead the evaluation
process to reward programmes that reproduce existing models of education, rather than
innovate.

Key recommendations

1. Adjust the weighting of evaluation criteria in assessment of new courses to
focus more on relevance, training and continuous improvement

The OECD review team considers that the current evaluation process for new courses could
be improved by adopting the following modifications:

e Include a separate section in the evaluation fiche on the relevance of the programme
to national development needs, taking into consideration the development of new
scientific areas and the knowledge and skills required for the further development
of the private and public sectors in the country, including in natural sciences, social
sciences and the arts.

e Increase the weight attached in the evaluation of new courses to the training
dimension of programmes and support provided to students, with an assessment of
the likely capacity of the programme to equip students with relevant research and
transversal skills.

e Include a more explicit requirement for a programme development plan for all new
programmes approved, setting out specific and measurable goals over time.

2. Bring additional perspectives into the evaluation of new programmes

To bring a broader range of perspectives to the process and potentially promote innovation
and inter-disciplinary cooperation, CAPES should involve one or more academics from
other academic fields in the field committees undertaking the assessment of new courses.
In addition, to bring in expertise and perspectives from outside the academic community,
CAPES should consider appointing specialists in economic development and the evolution
of skills and knowledge requirements, as well as representatives of the private economy
and the wider public sector to the Scientific and Technical Council (CTC-ES). If
implemented effectively, this could ensure that final decisions on programme approval take
into account broader national needs and developments.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

38 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

3. Maintain programme-level accreditation in the medium-term, but consider the
long-term desirability of transitioning to institutional self-accreditation for
established institutions and programmes

Brazil’s postgraduate education system has grown rapidly in recent years and might still be
considered to be in a phase of consolidation, when compared to postgraduate education
systems in many other OECD and partner countries. In the medium term, it therefore makes
sense to maintain course-level accreditation, to maintain oversight of the continued
development of the system and ensure the promotion of quality. In the longer term, it could
be possible to move to a system of institutional self-accreditation linked to a strengthened
model of institutional accreditation. This would allow universities to start academic
postgraduate programmes if they met certain criteria in terms of staff and profile and had
been judged to have strong institutional quality systems in an institutional quality review
(see below). The provision of publicly funded scholarships and additional programme
funding should certainly remain dependent on positive external evaluation of the
programme, in line with practice in many OECD systems.

4. Clarify the objectives of periodic evaluations and rebalance the focus of
evaluation criteria to include greater focus on student outputs and outcomes

The periodic (four-yearly) evaluations of postgraduate programmes currently devote
disproportionate attention and resources to assessing the outputs of academic staff. CAPES
evaluations should focus on assessing the conditions for and performance of postgraduate
training, not the research output of academic departments. The OECD review team
therefore recommends increasing the weight attributed to educational processes, student
outputs and employment outcomes, and reducing the weight attributed to staff outputs. This
would make it possible to reduce the time and resources allocated to assessment of staff
output, by assessing only a limited sample of research output. The Qualis system for journal
rankings should also be reviewed, to introduce more uniformity in the classification of
journals between knowledge fields. CAPES should also consider whether it is feasible
systematically to include interviews with course and programme coordinators as part of the
periodic assessment of courses and programmes, to gain additional insights into the
operation and performance of the programme and answer questions arising from
documentary evidence.

5. Ensure those judging whether programmes are of international standing really
have an international perspective

Given Brazil’s aspiration to develop a world-class postgraduate training system, it would
be valuable to gain an international perspective on the programmes judged nationally to be
among the best in the country. The OECD review team therefore recommends that CAPES
systematically involve non-Brazilian academics in the assessment of programmes pre-
selected by field committees as candidates for being programmes of international quality
or excellence. In light of the number of programmes involved, it is likely to be most feasible
to concentrate this international involvement on programmes proposed for the top score of
seven. It may be possible to organise international peer-review committees, who are able
to review synthesised information about the programmes under review in English or
Spanish, and potentially conduct group interviews remotely or in person with programme
coordinators.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 39

6. Undertake evaluations of specific components of the CAPES system and aspects
of academic postgraduate provision as inputs to future policy

The OECD review team identified two specific issues where further information and
analysis appears to be required in order to plan future policy for academic postgraduate
education in Brazil, and its external quality assurance:

e First, the full costs associated with the current system of external peer review are a
“black box”. Peer review is inherently time-consuming and therefore expensive.
The time academic staff spend involved in peer review is time they are not
dedicating to their core activities of teaching, research and engagement with
society. In order to help plan the future development of the system of peer review,
CAPES should undertake an assessment of the cost of the time used by members
of the field committees in the evaluation process, including the unit cost per
programme evaluation.

e Second, there is a wider question relating to the future of academic (stricto sensu)
master’s programmes. It would be valuable to undertake a systematic evaluation of
the role of master’s education in Brazil, including a specific focus on the profile
and effectiveness of the Professional Master’s programmes created in recent years.
This evaluation should consider, in particular, the destinations of previous
graduates from these programmes and the views of the academic community and
private and public sector employers on the relevance and future role for master’s-
level education in Brazil.

1.6. Assuring the quality of higher education institutions
Main findings

HEIs are also subject to monitoring and periodic re-accreditation

Legally, both private and federal public institutions are subject to periodic re-accreditation
(recredenciamento), based on on-site reviews coordinated by INEP. For private
institutions, successful re-accreditation is a prerequisite for their continued operation
(although “de-accreditation” is rare). For federal public institutions, the process is
essentially no more than a formality, as they cannot have their accreditation removed. The
period for which (re-)accreditation is valid varies depending on the organisational status of
the institution and its institutional quality score (CI). Universities and university centres are
only re-accredited every eight to ten years, while colleges must be re-accredited at least
every five years. In addition, institutions are subject to annual monitoring, based on the
average performance of their programmes in relation to SINAES programme-level
indicators and the results of CAPES evaluations for stricto sensu postgraduate programmes.
The weighted averages of the Preliminary Course Score (CPC), and, where applicable, the
scores attributed by CAPES for new and existing postgraduate programmes, are used to
produce an overall score for each institution called the “General Course Index” (IGC).

But institutional review plays a far less prominent role than in many other systems
of external quality assurance

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

40 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

cycle. For institutions that perform poorly in ENADE and on the CPC, this leads to regular
programme-level inspections, using prescriptive processes that limit the room for
manoeuvre for institutions. For institutions that tend to perform well in relation to ENADE
and the CPC, particularly universities and university centres that are only subject to
institutional review every eight to ten years, on-site evaluations by external reviewers are
comparatively infrequent occurrences.

There are few incentives for institutions in this position to develop strong internal quality
assurance systems that go beyond the minimum requirements imposed by the legislation,
or to promote quality enhancement internally on a continual basis. Interviews conducted
by the OECD review team in several institutions suggest that Internal Evaluation
Commissions (CPAs) focus primarily on ensuring compliance with SINAES rules and
delivering data to INEP, rather than developing internal quality systems tailored to
institutional needs or promoting innovation and quality improvements. This contrasts with
the situation in many European countries and in the United States, where institutional
review and evaluation of internal quality procedures form the core of many external quality
assurance practices.

The General Course Index (IGC) provides limited signals about institutional
quality

The IGC score — also calculated on scale of one to five - is used by external bodies and the
media in reporting about the quality of higher education in Brazil. The IGC is widely
perceived as a visible public signal of institutional quality that institutions themselves
feature in advertising. The real signal value of the IGC as a quality indicator for consumers
is limited, however. While IGC scores range, in principle, from one to five, scores of one
are virtually unknown, and nearly all scores cluster at values of three and four. In 2016,
93% of universities and 96% of university centres received scores of three or four. Setting
aside the validity or reliability of the IGC, it is clear that its discriminating power for non-
college institutions is low. Although the reputational effects of the IGC can be important,
it is not an indicator that is likely to have an impact of how institutions understand and
manage the quality of the education that they provide. The IGC does not introduce new
performance information for institutional leaders.

On-site re-accreditation reviews do not consider evidence of institutional
performance and attach little weight to the quality of internal quality processes
and their practical implementation

Like the other on-site review processes (such as recognition), the re-accreditation review
process coordinated by INEP (potentially only every ten years) is focused on input and
process, rather than outputs or performance, and reviewers are responsible for scoring
qualitative indicators on a five-point scale. Given that the process of re-accreditation
necessarily focuses on institutions that are already operating, with graduating students and
graduates, there is scope to include greater consideration of outputs (graduates and
evidence of their learning outcomes) and outcomes (graduate destinations) in the
institutional assessments at this stage. The current evaluation instrument for institutional
re-accreditation devotes comparatively little attention (in terms the number of indicators
and judgement criteria) or weight to assessment of the internal evaluation capacity of
institutions.

Owing to the schedules for re-accreditation, the institutional quality score awarded through
re-accreditation processes (CI) is not calculated and reported on an annual basis, but rather

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 41

with a periodicity that may range from three to ten years. In light of its infrequency, and
perhaps because it is not linked to student outcomes as observed in ENADE, the CI score
appears to function solely as a regulatory input, and not as a public signal of institutional
quality.

Key recommendations

1. Reduce the period of re-accreditation for universities and university centres

Universities in some of the best-regarded higher education systems in the world must
undergo external institutional reviews every four, five or six years. This is the case in the
United Kingdom, the Netherlands and Sweden, for example. The current eight or ten-year
accreditation periods for universities and university centres mean these institutions have
few incentives to develop robust institutional quality mechanisms and problems in
institutional quality management may go undetected for long periods. Instead of the current
system, institutions with demonstrated internal quality capacity could be rewarded through
dispensation from some or all aspects of programme-level review, subject to successful re-
accreditation on a five or six-year cycle (see below).

2. Reduce the weight attached in institutional re-accreditation reviews to input
and process indicators that measure basic supply conditions for higher education

There is scope to rebalance the weights attributed to the evaluation indicators used at the
stage of institutional re-accreditation, away from inputs and towards processes and outputs.
A first aspect of this is to remove indicators that measure basic supply conditions for higher
education, such as infrastructure and equipment and general management policies. The
availability of suitable infrastructure to supply each undergraduate programme is verified
through the programme-level recognition and renewal of recognition processes, while some
of the most general institutional policies are unlikely to change — or need to change -
considerably over time. It is therefore wasteful to devote resources to re-evaluating and re-
scoring these kinds of variable through the re-accreditation review. The inclusion of these
indicators also reduces the proportional weight attributed to factors that are important to
verify in re-accreditation such as educational results and institutional performance.

3. Increase the weight attributed to outputs and outcomes

Evidence about educational results and institutional performance is neglected in the current
system of institutional re-accreditation. While processes of accreditation cannot take into
account programmatic and institutional performance, re-accreditation can — but does not.
Institutions should be able to graduate most students who begin their studies, and they
should do so in a timely way. Those who graduate should be able to find employment,
preferably in fields related to their area of study — and most certainly so if their studies have
a career orientation — whether accounting, civil engineering, or nursing.

Quantitative programme and institutional indicators should ideally focus on the outputs and
outcomes of higher education, while on-site reviews conducted by peers would helpfully
focus on the inputs and processes that generate the outputs and outcomes observed in
indicators. For example, indicators focused on outputs or outcomes, such as graduation
rates, would be complemented by an on-site review process that examines the conditions
that affect variation in these rates. These conditions include student advice and mentoring
processes; how institutions identify students at risk of falling behind or dropping out; and
the social or psychological, and academic support services provided to students at risk.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

42 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

4. Increase incentives for institutions to take a strategic view of quality

The processes of institutional quality assurance do not encourage institutions to take a truly
strategic and institution-wide view of quality. The IGC generates a score that is an
aggregation of programme-level results. However, it does not generate a score that has been
demonstrated to be useful in differentiating different levels of institutional performance or
providing actionable feedback to institutions. Institutional Development Plans (PDIs), in
their current form do not appear to provide an opportunity for institutions to take a
comprehensive and strategic view of their institution, its profile, and the quality of its
educational programmes. It would be valuable to provide incentives to institutions to
develop more meaningful PDIs with a stronger focus on how quality across a range of
dimensions can be maintained and enhanced. One way to do this is to make assessment of
internal quality policies and practice a much bigger part of the re-accreditation process,
through greater weighting in the relevant evaluation instrument for on-site reviews.

5. Move to a system where institutions that can demonstrate strong internal
quality assurance capacity and a proven record of delivering quality can accredit
(authorise and recognise) their own programmes

Finally, processes for demonstrating institutional quality do not permit higher education
institutions to demonstrate that they have the capacity to take care of quality, and should
be authorised to act as self-accrediting organisations, and should be permitted to create,
revise, and eliminate programmes on their own initiative — as happens in other higher
education systems in the world. The process of re-accreditation — specifically, the resulting
CI score — changes marginally the periodicity of institutional reviews, but it does not alter
the level of responsibility that institutions are permitted to exercise. If account for
institutional quality is to be joined up to institutional responsibility for the quality of
programmes, it will need to be a very different and more robust process than at present.
Examples of such differentiated models — where some institutions are subject to
programme-level review and others are accorded self-accrediting status on the basis of
rigorous institutional review - exist in other systems of higher education and could serve as
inspiration for Brazil.

1.7. Governance of external quality assurance
Main findings

The current governance landscape for quality assurance, involving SERES,
CONAES, INEP and CAPES has some strengths

There are important strengths to the governance and implementation of quality assurance
in Brazil. For example, INEP is recognised internationally as a leading public agency for
educational assessment. Its wide experience with large-scale assessment and its capacity to
manage data collection systems provides the nation’s higher education quality assurance
system with a high level of competence. CONAES has succeeded in attracting experts to
its council, and through them has been able to mobilise higher education research from
across the nation to inform the further development of SINAES.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 43

The basic legitimacy of external quality assurance is not questioned and the
system has developed significant experience and capacity in evaluation

The basic legitimacy and integrity of the quality assurance system is widely accepted across
the higher education system, by public and private institutions alike, and by representatives
of academic staff and the administrators and owners of higher education institutions. In the
course of its implementation, SINAES has used a range of evaluation techniques —
including self-assessment, peer review, and external review grounded in student
assessment - that has been widely welcomed. Moreover, some higher education institutions
in Brazil now closely monitor the experience of their students and their readiness to
participate in external assessments. Others are making efforts to use compulsory self-
assessment and peer-review processes as opportunities for improvement, and to engage
broadly their university community in the assurance of quality.

...but the current system of governance faces three main challenges

There are three fundamental challenges facing the institutions of quality assurance that
merit attention and improvement.

1. First, the design of quality assurance institutions creates conflicting responsibilities
for the Ministry of Education. MEC establishes, funds, and steers the federal
university system, through its Secretariat for Higher Education (SESu). At the same
time, it is responsible, through SERES and, indirectly, INEP, for evaluating their
performance and for regulatory actions concerning the programmes they offer.
These conflicting responsibilities lead the nation’s higher education institutions,
especially its private institutions, to view the Ministry as a champion of one sector,
rather than a neutral arbiter among all.

2. Second, while CONAES is responsible for providing guidance and feedback on the
functioning of SINAES, it is not properly resourced and organised to do so.
CONAES does not have its own professional staff or a dedicated budget, and lacks
the capacity to undertake the sort of detailed and sustained analytical work that is
needed to evaluate how SINAES is working. Instead, it depends upon the input of
implementing bodies whose work it is to supervise and guide, most especially
INEP. This dependence is exacerbated by the participation of the implementing
bodies on the council itself. It lacks sufficiently wide input — from professional
bodies, employer associations, and other centres of government - to take into
account the broader social responsibilities of higher education.

3. Finally, in most higher education systems, responsibility for promoting and sharing
quality improvement practices lies with bodies outside of government - with
associations that represent sub-sectors (such as research, confessional, or
polytechnic universities), and with bodies that represent professional groups within
higher education institutions. The review found few examples of the engagement
of equivalent bodies in Brazil in research, advocacy, and training in support of
quality improvement, and little attention on the part of public authorities to their
potentially important role.

The federal system of quality assurance does not apply to all higher education
providers in Brazil

As noted earlier, the systems for external quality assurance of HEIs and undergraduate
programmes analysed in this report apply only to private HEIs and federal public HEIs.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

44 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

State and municipal public institutions — which account for almost 10% of enrolment - are
not subject to SINAES, but rather to state-level regulatory and quality assurance rules.
Although this situation reflects the constitutional distribution of competences in the
Brazilian state, which allows considerable autonomy to states and municipalities, it leads
to a fragmented system and means there is no single national benchmark of higher
education quality. A single quality reference framework would make external quality
assurance for higher education more transparent and understandable for students and their
families.

Key recommendations

1. Create an independent quality assurance agency

To address the conflicting responsibilities of MEC — or indeed any future ministry
responsible for higher education - Brazilian authorities should consider creating an
independent quality assurance body that stands outside the Ministry, in line with practice
in many OECD and partner countries. This agency would take the lead in implementing
the reformed system of quality assurance proposed in this report. Good international
models of bodies with strong legal, financial, and administrative independence exist. In
systems with a similar legal tradition to Brazil, such agencies include, for example,
Portugal’s Agency for the Assessment and Accreditation of Higher Education (A3ES).

The work to design and create any new agency for quality assurance in Brazil will need to
address some key questions:

Which existing functions should be transferred to the new agency? In principle, the new
agency would combine the evaluation functions coordinated by INEP’s higher education
evaluation directorate (DAES) and the regulatory and supervisory roles of SERES. The
changes to the overall model of regulation, evaluation and supervision proposed in this
report — such as increased focus on institutional review, reduced numbers of programme-
level reviews, a reformed ENADE and a new indicator dashboard - will affect requirements
for staff in different roles. The advantages and disadvantages of creating specific evaluation
units for different sets of disciplines (natural sciences, social sciences etc.) should be
considered. Such units, integrated within the agency, could potentially allow evaluation to
be better tailored to individual disciplines and work more closely with the discipline-
specific CAPES evaluations.

Should some tasks be devolved to decentralised offices in the states? The current system of
quality assurance in the federal higher education system is highly centralised, with all
evaluation and regulation activities coordinated from Brasilia. Devolving responsibility to
regional departments might theoretically allow a more differentiated approach to quality
assurance, with better consideration of the large regional differences in Brazil. However,
in the view of the OECD team, distinct quality assurance procedures in different parts of
the country would risk creating a two- (or multi-)tier system and undermining national
recognition of quality standards. It could be possible, however, to establish regional offices
to house professional inspectorates to undertake inspection of infrastructure and
institutional management, freeing academic peer reviewers to focus on assessment of
academic performance, potentially remotely (see above). The costs of the current system
of peer review and the potential costs of a permanent inspectorate would need to be assessed
in detail.

How should the new agency be funded? The current system of external quality assurance
in Brazil is funded by a combination of public resources (paying the salaries of public

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 45

servants, for example) and fees paid by institutions for evaluation activities. Quality
assurance agencies in a number of systems, including the Portuguese example mentioned
above, are funded primarily through fees from institutions. To ensure efficient use of public
resources, this should be the long-term aim in Brazil. A thorough analysis will be required
to determine the costs of a new agency and the level of fees needed to finance its operation.

The OECD team recognises that there is an existing proposal to create a National Institute
for the Supervision and Evaluation of Higher Education (INSAES), that was introduced as
a draft bill in Congress in 2012 (Congreso Nacional, 2012;4)), but not pursued. This
initiative effectively also proposed a merger of the functions of SERES and INEP, but was
criticised for its potential cost and limited added value. The OECD team believes that a
new agency would be the most effective way to implement a reformed system of external
quality assurance. The reforms proposed in this report are vital to improve the effectiveness
and efficiency of the system and any future agency must be designed to operate as
efficiently as possible and with limited direct public subsidy.

2. Strengthen CONAES

To ensure that the quality assurance agency has an advisory council that brings a wide
social vision to its work, CONAES could take on this responsibility, after substantial
modification. CONAES would be a council with members holding fixed and staggered
terms to ensure their independence of government, and encompass balanced representation
from students, public and private sector employers, instructors from public and private
higher education institutions, higher education administrators, leading researchers, and the
senior policy official in MEC with responsibility for taking a comprehensive view of higher
education.

3. Restructure the government departments that are responsible for higher
education

MEC -— or any future ministry responsible for higher education - can support the
improvement of quality assurance by restructuring its responsibilities for higher education.
This could entail creating a post for a principal policy officer who takes a comprehensive
and strategic view of the entire Brazilian higher education system — which the Ministry
presently lacks. Units organised along sectoral lines, for example, could support the work
of a senior official. These might include groups responsible for (a) federal universities; (b)
private universities; (c) technical higher education; and (d) coordination with state and
municipal higher education institutions. This scheme of organisation would benefit the
nation’s quality assurance system by supporting a strategic and comprehensive vision for
the higher education system, by clarifying the role of private provision within the system,
and by encouraging continued differentiation of institutions and policies.

4. Incentivise the development of expertise in quality assurance in sector
organisations

In monitoring and evaluating the nation’s quality assurance system, a reconstituted quality
assurance agency and advisory council (i.e. CONAES) should focus on supporting the
development of quality enhancing organisations outside of government. For example, it
could support collaboration among state and national bodies of institutional evaluation
offices (CPAs), so they share experiences of quality management and improvement
practices with one another.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

46 | CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS

5. Explore how a reformed external quality assurance system could also apply to
state and municipal institutions

A single system of external quality assurance applying to all higher education institutions
in the country would be more transparent for students and the public than that current co-
existence of a large federal system and individual systems for state and municipal
institutions in each state. The federal and state authorities, working with the higher
education sector, should explore how — and under what conditions - a reformed federal
system of quality assurance could be applied to state and municipal institutions, while
respecting the distribution of competences enshrined in the constitution of the Union.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 1. ASSESSMENT AND RECOMMENDATIONS | 47

References

Congreso Nacional (2012), Projeto de lei: cria o Instituto Nacional de Supervisao e Avaliagao [4]
da Educagao Superior - INSAES, e da outras providéncias., Secretaria de Assuntos
Parlamentares, http://www.planalto.gov.br/ccivil_03/Projetos/PL/2012/msg398-
3 lagosto2012.htm (accessed on 30 November 2018).

Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review”, NBER Working [2]
Paper Series, No. 23744, National Bureau of Economic Research, Cambridge,
http://www.nber.org/papers/w23744 (accessed on 23 November 2018).

Presidéncia da Republica (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional [1]
de Avaliagao da Educagao Superior — SINAES e da outras providéncias (Law 10 861
establishing the National System for the Evaluation of Higher Education - SINAES and other
measures), http://www.planalto.gov.br/ccivil 03/ ato2004-2006/2004/lei/110.861.htm
(accessed on 10 November 2018).

TCU (2018), RA 01047120170 Relatorio de auditoria operacional. Atuagao da Secretaria de [3]
Regulagdo e Supervisdo da Educagao Superior do Ministério da Educagdao - Seres/Mec e do
Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira - Inep nos processos
de regulagao, supervisdo e avaliagao dos cursos superiores de graduagao no pais.

(Operational audit report: Activities of SERES/MEC and INEP related to regulation,
supervision and evaluation of higher education programmes in the country), Tribunal de
Contas da Uniao, Brasilia, https://tcu.jusbrasil.com.br/jurisprudencia/5829 15007/relatorio-de-
auditoria-ra-ra-1047120170/relatorio-582915239?ref=juris-tabs (accessed on

13 November 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 49

2. Scope of the review and analytical framework

This chapter provides a brief overview of the main aspects of the external procedures in
place in Brazil to assure the quality of the federal higher education system - the subject of
this review - before setting out the framework that the review team has used to structure
and guide its assessment of the relevance, effectiveness and efficiency of these procedures.
To contextualise the analysis in the review and the analytical framework used, the chapter
also provides a brief review of some of the main developments and challenges faced by
higher education quality assurance systems internationally.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

50 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

2.1. Focus of this chapter

This chapter provides a brief overview of the main aspects of the external procedures in
place in Brazil to assure the quality of the federal higher education system, before setting
out the framework that the review team has used to structure and guide its assessment of
the relevance, effectiveness and efficiency of these procedures.

As with other OECD education policy reviews, this review provides a qualitative
assessment of the specific policies under scrutiny. It takes into account the objectives
established by national authorities for the policies in question and bases its judgements on
documentary evidence and stakeholder opinion regarding the implementation of the
policies in Brazil; lessons from international standards and practice in other OECD and
partner countries; and the experience and expert opinion of the review team members.

The review recognises that systems for the external quality assurance of higher education,
like other types of public policy, need to be tailored to the specific situation in the
jurisdictions where they are applied. As in other policy fields, there is no single set of “best
practices” in the external quality assurance of higher education that can be applied
uniformly to all higher education systems. This review therefore presents international
standards and guidelines - to the extent that these exist - and practice examples from other
countries as reference points and potential sources of inspiration for Brazil, rather ready-
made models that could or should be applied in the Brazilian context.

The simple analytical framework used to guide the review - outlined more fully later in this
chapter - is based on standard policy evaluation criteria. The most important criteria are the
relevance of the objectives of different parts of the external quality assurance system to the
challenges and requirements of the Brazilian context; the effectiveness of the different
aspects of the system in achieving their objectives; and the efficiency (and cost-
effectiveness) with which they do this. In judging the relevance, effectiveness and
efficiency of different aspects of the quality assurance system for higher education in
Brazil, the review team has taken into account domestic criteria (for example, does the
system fulfil the objectives established by Brazilian legislators? Is the system considered
to be effective by Brazilian stakeholders?); and international criteria (for example, does
the Brazilian system promote minimum quality standards, differentiated assessment of
quality or quality enhancement as well as systems in other jurisdictions?).

Before setting out in more detail the analytical framework and the evaluative questions that
structure the rest of the analysis, the following sections first provide an overview of the
main components of Brazil’s external quality assurance systems for higher education and,
second, briefly examine the development of external quality assurance systems in higher
education more generally and common challenges faced internationally.

2.2. External quality assurance of higher education in Brazil

The federal government regulates most higher education in Brazil

Brazil has well-established systems in place at national level to regulate the operation of
private and public higher education providers in the federal higher education system and
assess and monitor the quality of their teaching and learning activities.

Within Brazil’s federal governance structure, responsibility for providing and regulating
higher education is formally shared between the federal government, the 27 federative units
(the 26 states and the federal district of Brasilia) and the municipalities. The federation,

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 51

many states and a small proportion of (large) municipalities all have public higher
education institutions (HEIs) falling under their responsibility. All private higher education
providers in the country legally fall under the regulatory responsibility of the federal
government. The “federal higher education system” thus comprises federal public and all
private HEIs. Of the roughly 2 400 HEIs in Brazil, 92% (federal public and private) fall
under the regulatory responsibility of the federal government and these institutions,
together, account for 91% of undergraduate enrolment. Three quarters of total
undergraduate enrolment in Brazil is in the private sector (see Section 3.4 for an overview
of the Brazilian higher education landscape).

Quality is assured through related processes of regulation, evaluation and
supervision

The federal authorities assure the quality of higher education institutions and undergraduate
education! in the federal system through a combination of distinct, but closely related,
processes referred to as regulation, evaluation and supervision and currently coordinated
by the Ministry of Education (MEC):

The regulation of higher education, undertaken by Secretariat for Regulation and
Supervision of Higher Education (SERES), a division of MEC, involves issuing formal
approval (in the form of regulatory acts) for the operation of higher education institutions
and individual undergraduate programmes. All private and public HEIs in the federal
system formally require accredited status to operate? (and periodic re-accreditation) and
official recognition of the undergraduate programmes they provide. Programme
recognition must also be renewed periodically, based on the results of quality evaluations.
As discussed in Chapter 4, depending on their level of institutional autonomy, institutions
may also need prior authorisation from MEC to start new undergraduate programmes.

SERES makes its decisions regarding the accreditation and re-accreditation of HEIs and
the authorisation, recognition and renewal of recognition of undergraduate programmes
taking into account the results of institutional and programme evaluations, undertaken by
the evaluation directorate of the Anisio Teixeira National Institute for Educational Studies
and Research (INEP). These evaluation activities encompass external assessment of
institutions and programmes and assessment of student learning outcomes through the
National Examination of Student Performance (ENADE). Collectively, these three types
of evaluation (institutional, programme and student) form the National System of Higher
Education Evaluation (SINAES), which was established in its current form in 2004
(Presidéncia da Republica, 2004;1;). The evaluation of institutions and programmes
(discussed in more depth in the following chapters) is based on on-site inspections by
external review panels, results obtained by graduates in ENADE and, at present, a limited
number of other quantitative indicators intended to measure programme quality.

Alongside its regulatory duties (the issuing of regulatory acts), SERES is also tasked with
the supervision of quality in the federal higher education system. In practice, this means
ongoing monitoring of quality levels in the system, using the results of the evaluation work
coordinated by INEP, and taking preventive and corrective measures when quality
problems are identified in individual programmes and institutions (Presidéncia da
Republica, 2017, pp. 1, art.2,2)). SERES can require institutions to take steps to address
quality problems identified in evaluations, impose sanctions or proactively require
intensified monitoring and evaluation of particular programmes or institutions.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

52 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

Figure 2.1 illustrates the relationship between regulation, evaluation and supervision,
whereby the evaluation activities, coordinated by INEP, that make up the SINAES, feed
into the regulatory and supervisory activities undertaken by SERES.

Figure 2.1. Regulation, evaluation and supervision in Brazilian higher education

Regulation
(SERES)

Evaluation

(INEP)

Supervision
(SERES)

Source: OECD Secretariat based on (Presidéncia da Republica, 2004,1); Presidéncia da Republica, 2017,2)).

Separate procedures exist to assure the quality of academic postgraduate
programmes

Postgraduate education in Brazil? takes the form of either purely vocational “specialisation”
programmes, referred to in the Brazilian system as Jato sensu programmes (which include
MBAs), or academic master’s, Professional Master’s and doctoral programmes, which are
classed as stricto sensu postgraduate programmes. Accredited HEIs in the federal education
system may provide /ato sensu programmes without regulatory authorisation or recognition
from MEC, provided they operate at least one formally recognised undergraduate
programme or at least one approved stricto sensu postgraduate programme (Presidéncia da
Republica, 2017, p. 29.2;)). The provision of academic, stricto sensu, postgraduate
programmes in conditioned on prior evaluation and approval by the Foundation for the
Coordination and Improvement of Higher Level Personnel (CAPES), a decentralised
agency of the Ministry of Education.

CAPES has been responsible for assuring the quality of academic postgraduate educational
programmes since the mid-1970s. Since 1998, it has operated a quality assurance system
that requires all proposed new academic postgraduate programmes to achieve a positive
evaluation score in a peer-review process and, from then on, to receive a positive evaluation
in periodic reviews (currently every four years). CAPES is responsible for the quality
assurance of all academic postgraduate programmes in Brazil, including in state and
municipal institutions, which are not subject to SINAES. In practice, it fulfils the roles of
evaluation, regulation and supervision for academic postgraduate programmes - functions
that are split between INEP and SERES for undergraduate provision in the federal system.

However, CAPES only evaluates individual postgraduate programmes. It requires that all
programmes within its remit be provided in formally accredited institutions and thus relies

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 53

on SERES and state education authorities to undertake institutional accreditation (for
institutions in the federal and state systems, respectively). Moreover, the results of CAPES
evaluations are taken into account in a composite indicator of institutional quality used by
INEP (see Chapter 7).

The normative framework for SINAES has recently been updated

The legal basis governing higher education in Brazil is provided in the federal constitution
and by the 1996 Education Act (Presidéncia da Republica, 1996,3}), which establishes basic
principles concerning the role of higher education, the division of competences between
the Union and the states and the role of the federal government in quality assurance.
SINAES was established by a specific law passed by Congress in 2004 (Presidéncia da
Republica, 2004;1;). This sets out the basic principles of the SINAES, including the
requirement for institutional, programme and student assessments, the role of on-site
inspections and ENADE, and the relationship between evaluation, regulation and
supervision. The current regime for evaluation of stricto sensu postgraduate provision was
established through ordinances (portarias) issued by MEC and CAPES from 1998 onwards
and which have been updated periodically.

The detailed implementation rules for SINAES (evaluation) and the processes of regulation
and supervision, reflecting the 2004 law, were initially established in a 2006 presidential
decree (Presidéncia da Republica, 2006;4}) and supplemented by ordinances issued directly
by MEC. This decree was replaced at the end of a 2017 by a new decree setting out the
processes of evaluation, regulation and supervision in greater detail than had previously
been the case (Presidéncia da Republica, 2017,2)). The changes brought about by this new
decree are discussed in the relevant sections of this report. In broad terms, however, it seeks
to simplify the administrative processes related to authorisation of undergraduate
programmes and modifications to programmes, rationalise procedures for on-site
inspections and provide more clarity about the preventive and corrective measures SERES
may take in its supervisory role (MEC, 20175).

These recent changes have been reflected to the extent possible in the review findings and
taken into account in the formulation of recommendations. However, as the changes have
only recently started to be reflected in the practices of SERES and INEP, it has not been
possible to seek views from stakeholders about their practical impact or to judge their
effectiveness in practice.

2.3. Quality assurance in higher education internationally: developments and
challenges

External quality assurance in higher education has developed in recent decades

The development of quality assurance systems in higher education is a comparatively
recent phenomenon. Historically, the quality of learning and teaching in higher education
was generally - and largely implicitly - assumed to be guaranteed by the presence of
academics with an established record of scholarship. A high degree of individual academic
autonomy in universities meant that, in many countries, neither public officials nor
university and faculty management frequently intervened in the teaching activities of staff
members. Although governments may have granted or withdrawn permission for
universities to operate, and may have provided funding, owned buildings and even
regulated staff conditions and salaries in public institutions, in much of the world they
rarely or never engaged with institutions’ day-to-day teaching and research activities. This

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

54 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

pattern of strongly independent public or private not-for-profit institutions is the traditional
model of higher education institutions in most of Latin America.

Over the last three decades, this situation has evolved, as governments across the world
have introduced external quality assurance systems for higher education and higher
education institutions have increasingly developed formalised internal quality procedures
for learning and teaching. Key factors driving increased government intervention have
included the expansion of higher education and the breakdown of historical trust
relationships among small elites; the expansion of demand-absorbing private sector
provision - particularly in Latin America and Eastern and Southern Europe; and a broader
trend for governments and society to demand greater evidence of performance and value
for money from publicly supported institutions and services (OECD, 2008;6); Brunner and
Miranda, 2016;7)).

Quality assurance systems have different objectives

Literature on quality assurance in higher education often distinguishes between the related
goals of accountability and quality enhancement (ESG, 2015;8;; CHEA, 2016)9)).
Accountability refers to the aim of providing information to assure the public (including
students and their families) of the quality of higher education institutions’ activities. This
is a goal common to virtually all external quality assurance systems in higher education.
Quality enhancement involves providing advice and recommendations on how higher
education providers might improve what they are doing, and is a less well-established
aspect of many external quality assurance systems.

Some external “quality assurance” systems are essentially little more than licensing
systems, where higher education providers are authorised to operate by public authorities
if they meet minimal operating requirements. Such systems provide only a minimum level
of accountability, which is essentially limited to providing the public with information on
whether a provider is legally registered or not. Other systems make authorisation and
accreditation of higher education activities (or the allocation of public funds) dependent on
positive results from a more in-depth evaluation of indicators of quality. Depending on the
relevance and quality of the criteria and data sources used by HEIs and external evaluation
agencies, such systems might be expected to offer a greater guarantee of minimum quality
standards and, as a result, a better degree of accountability. In addition to providing these
kinds of accountability guarantees, the most advanced external quality systems also seek
to promote enhancement of quality and continuous improvement. Such systems generally
seek to move beyond external regulation and control to promote a quality culture in all
areas of higher education activity, in partnership with HEIs and academic staff.

Defining and measuring quality is challenging

Arriving at a shared understanding of what quality in higher education is and how it should
be measured has proved challenging for those involved in developing and running quality
assurance systems (CHEA, 201 6j9)). There are at least three main reasons for this:

1. As noted in the 2015 Standards and Guidelines for Quality Assurance in the
European Higher Education Area “Higher education aims to fulfil multiple
purposes” and “stakeholders, who may prioritise different purposes, can view
quality in higher education differently’. The importance attached to different
aspects of the educational process or different kinds of learning outcome acquired
by students (for example, theoretical knowledge as opposed to practical skills) can

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 55

vary between individuals and groups within a single higher education system and
internationally.

2. The expansion of higher education has led to more diverse modes of provision and
student populations, and to demands for a more diverse set of programmes and
institutional profiles. This means that quality in higher education now comes in
more diverse forms and needs to be measured in more diverse ways. The co-
existence of higher education institutions with diverse missions and types of
programme is explicitly acknowledged in the Brazilian legislation establishing the
SINAES (Presidéncia da Republica, 2004, p. art.3;1)).

3. Even when there is agreement on the components of quality in different types of
higher education context, it may be conceptually and technically difficult to
measure these components in a reliable way. If what is important may not always
be measurable, what is measurable may not always be important. In the absence of
reliable or feasible direct measures of different aspects of quality, quality assurance
systems frequently resort to the use of proxy measures, which themselves can
become the subject of disagreement.

Quality education is education that is fit for purpose

Notwithstanding these challenges, recent international efforts to develop a shared
understanding of quality in higher education (CHEA, 2016, p. 48;9)) argue that quality
education is best conceived of in terms of fitness for purpose. In other words, good quality
higher education is education that:

1. Sets out to deliver the right kinds of learning outcomes for students - where the
right kinds of learning outcomes are ones that meet the needs of students and
society. The concept of learning outcomes encompasses both breadth and depth of
knowledge and skills: good quality education programmes establish the right
intended learning objectives, at the right level of complexity for their target student
population;

2. Creates and uses a learning environment (qualified teachers, teaching methods,
learning resources, opportunities to gain practical experience, etc.) suitable for
achieving these learning outcomes and;

3. Succeeds in practice in delivering the intended learning outcomes for as many
participating students as possible who begin studying.

The first element above is about intentions and, specifically, setting relevant learning
objectives. The second element concerns inputs (including teaching staff and resources)
and processes (including teaching methods and activities). The third element deals with the
output’ of the educational process. In addition, those concerned with the quality of higher
education may look at the broader outcomes of graduates who have gone through the
educational process and, in particular, their entry into and progression within the labour
market. Outcomes in this sense are influenced by the educational process, but also by a
range of other external factors.

Quality systems often focus on inputs and processes and less frequently on
outputs and outcomes

External quality assurance systems in higher education initially focused to a large extent
on measuring inputs, such as the level of qualification of teaching staff and the number of
books in the library, as a way of gauging the quality of provision. Although input measures
are usually readily available and objective, on their own they provide little evidence of

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

56 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

quality in practice. The fact that a member of teaching staff has a master’s degree of a PhD
might be considered important and perhaps necessary, but it is not sufficient to assure their
ability as a teacher. Funding is another input. Adequate funding might be assumed to be a
precondition for creating an effective learning environment. However, defining what level
of funding is “adequate” is often challenging and controversial and, once defined, adequate
funding is not, in itself, a guarantee of quality.

Processes - in particular learning, teaching and assessment methods - can provide an
indication of the likely effectiveness of the teaching and learning experience for students,
and are therefore monitored in many quality assurance systems in higher education. For
example, procedures for external marking of written assessments and examinations may be
taken into account as an indicator of the reliability of assessment processes (and thus the
validity of the results and qualifications awarded by an educational programme). However,
many aspects of teaching and learning are difficult to capture and assess in a binary (yes/no)
or quantitative way, which makes it difficult to collect quantitative data on such processes.

In recent years, there has been an increasing focus on the possibility of using output and
outcome information in quality assurance systems. The most direct outputs of the
educational process are graduates with increased knowledge and skills (learning outcomes)
acquired through their education. Isolating the specific added value of a student’s higher
education experience from other factors like their social and cultural background is
intrinsically very challenging. However, internationally, standardised tests are used in some
jurisdictions to measure the skills and competencies of higher education students and
graduates in a comparable manner.

In the United States, for example, the CLA and CLA- tests have been deployed widely to
test generic competencies (CAE, 2018 ;10)), while in Mexico, the Exdmenes Generales para
el Egreso de Licenciatura (EGEL) are used by many institutions to test graduates in specific
disciplinary areas (CENEVAL, 2018 ;11)). In Colombia, all higher education students are
required to take a general competency test (Saber Pro) in order to graduate (ICFES,
2018;12). Although the Saber Pro exams in Colombia are compulsory, the results obtained
by students are not used directly to generate quality scores for the institutions they attended.
As discussed in Chapter 5, the ENADE examination in Brazil is the only example, in a
major higher education system, of large-scale, external examinations that are both
compulsory for students and used directly in the quality assurance of programmes and
institutions.

Other output or outcome related measures do not look at student learning per se, but rather
at indirectly related issues. For outputs, this includes graduation and completion (e.g. the
proportion of students successfully completing their course). For outcomes, this includes
employment outcomes (e.g. the proportion of graduates that are employed and in what
types of job).

The relationship between output and outcome measures and course quality is not always
straightforward. Although a high-quality course may be expected to provide good support
to students from different backgrounds to allow them to complete their course successfully,
a 100% completion rate may be an indication of low standards, rather than good quality.
While it is logical to assume that high-quality courses prepare students well to get good
jobs, graduate employment outcomes depend on a wide range of factors beyond the quality
of the educational programme. Graduates from a prestigious, but objectively poor quality,
course may consistently succeed in getting good jobs. Graduates from high-quality courses
may have difficulty in finding appropriate work if the relationship between the course

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 57

content and knowledge and skills required in the labour market is weak or general
employment conditions are difficult.

Table 2.1. Quality indicators: objectives, inputs, processes, outputs and outcomes

Example indicators

Objectives of the Relevance / appropriateness of intended learning outcomes for the educational programme.
educational process Transparency / clarity of intended learning outcomes.
(Objective indicators) Intended learning outcomes have been developed taking into account latest developments in the

scientific or professional area concerned.

Intended learning outcomes have been developed in consultation with relevant external
stakeholders (employers, sector associations, unions etc.).

Inputs to the Funding available to ensure stable operation of the educational provider (continuity of service,
educational process framework conditions for educational provision).
(Input indicators) Funding available for the educational processes (to pay for staff, buildings, equipment etc.).

Ratio of teaching staff to students.

Qualifications and research outputs of teaching staff.

Surface area of classroom space per student.

Accessibility of teaching and learning spaces for students with disabilities.
Library facilities (number of books, online resources available, etc.).
Laboratory facilities.

The educational Teaching contact hours.
process (where relevant) Hours of practical activities built into curriculum (laboratory time etc.).
(Process indicators) (where relevant) Work-based learning experience built into curriculum and conditions of this

(internships, placements etc.).

Use of online / blended learning in curriculum.

Staff time dedicated to academic support outside class.

Availability of / time dedicated to pastoral / non-academic support.

Marking arrangements for assessments (double marking etc.).

Ongoing quality assurance measures in place (peer review of teaching etc.).

Outputs of the Proportion of students graduating successfully (successful completion rate).
educational process Average time taken to graduate.
(Output indicators) Knowledge and skills of students (learning outcomes) at end of programme.

Added value: additional knowledge and skills (learning outcomes) gained during the programme
(measured by tracking skills over time).

Outcomes of the Employment status of graduates.
educational process Alignment between qualification type and level of employment.
(Outcome indicators) Graduate earnings.

Social mobility.

Evidence of active citizenship among graduates.

Source: OECD, Education and Skills Directorate.

Measuring the quality of postgraduate education may present specific challenges. Although
the quality of taught master’s programmes is often assured using the same types of
indicators as for undergraduate education, the quality of research-oriented master’s and
doctoral programmes is also influenced to a great extent by the broader research
environment in which they take place. Assessment of the quality of postgraduate provision
may therefore consider research-related input, process and output indicators. These can
include the research performance of the academic department hosting the programme
(input), opportunities for postgraduate students to attend conferences or organise and
participate in research-related events (process) or the number and quality of research
outputs by students (output). However, in a world where postgraduate students increasingly
go on to work outside academia and scientific research, the quality indicators used need to
adapt to take this into account.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

58 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

2.4. An analytical framework for the review

The objectives of the review

The terms of reference for this review, agreed with the Ministry of Education at the start of
the project, call on the review team to assess the relevance, effectiveness and efficiency of
the external quality assurance procedures applicable to undergraduate and postgraduate
programmes and HEIs in the federal higher education system in Brazil. Specifically, the
terms of reference ask the team to consider the effectiveness and efficiency of the systems
in a) ensuring minimum quality standards (basic accountability); b) providing
differentiated measurement of quality (between types of provision and levels of quality
offered) and; c) promoting improvement of quality and quality-oriented practices in HEIs
(quality enhancement). The team was invited to provide an analysis in relation to these
points and recommendations for improving the systems in place.

Disaggregating the different components of the quality assurance system

As illustrated by the earlier discussion, the procedures for external quality assurance in the
federal higher education system comprise a set of distinct processes for HEIs,
undergraduate programmes and stricto sensu postgraduate programmes. For each of these,
specific procedures exist, governing:

1. “Market entry” for new institutions and programmes, based on ex-ante assessment
of the likely quality of the institution or programme proposed (before they begin
operation) and, for undergraduate programmes, on their quality after their first years
of operation (when the first student cohort has completed half the programme);

2. Ongoing monitoring and evaluation of quality for existing institutions and
programmes and;

3. The provision of feedback to programmes and institutions and actions to taken to
respond to quality problems detected through the evaluation processes, including
sanctions.

These three stages of quality assurance, which bring together the regulatory and
supervisory functions of SERES, the evaluation function of INEP and the combined
evaluation and supervisory roles of CAPES, are illustrated in Figure 2.2.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 59

Figure 2.2. External quality assurance system for higher education in Brazil

Components of the external quality assurance system for higher education in Brazil

Stricto Sensu Postgraduate
programmes

Undergraduate
programmes

(Private) Institutions

Market entry

Evaluation of new course proposals

Accreditation | Authorisation (some) & Recognition (all)

(credenciamento) (Autorizagéo) & (Reconhecimento) | (Avaliagao de PCN (CAP Rec Novos)
Ongoing monitoring & evaluation

Renewal of recognition Four-year evaluations
(renovagdo de reconhecimento) (Avaliagdo Quadrienal)

Feedback and responding to problems

Feedback provided + low scoring
Supervisory measures by SERES programmes lose diploma validity and
funding

Source: OECD Secretariat, based on Brazilian legislation and policy documents.

The review set out to analyse all of these components. Taking into account the way the
different components are organised in practice and related to each other, they have been
grouped and analysed as follows:

1. Market entry for new HEIs and new undergraduate programmes (SINAES
evaluation by INEP and regulatory decisions by SERES) is analysed in Chapter 4
of this report.

2. Ongoing monitoring and evaluation of existing undergraduate programmes and
related feedback and corrective measures (SINAES evaluation by INEP and
regulation and supervision decisions by SERES) are analysed in Chapter 5.

3. Market entry and periodic evaluation of stricto sensu postgraduate programmes and
related feedback and corrective measures, coordinated by CAPES are analysed in
Chapter 6.

4. Ongoing monitoring and evaluation of HEIs and feedback and corrective measures
(SINAES evaluation by INEP and regulation and supervision decisions by SERES)
are analysed in Chapter 7.

In addition, Chapter 8 of this report analyses the governance and administrative bodies and
arrangements that have been created to implement and oversee the processes outlined
above.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

60 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

An evaluative framework to structure the analysis

The terms of reference for the review specify basic evaluation criteria, which should
structure the assessment of the components of the higher education quality assurance
systems set out above:

1. The analysis of the relevance of the procedures and governance arrangements needs
to consider the extent to which the objectives of these procedures and arrangements
respond to the challenges and requirements of the Brazilian context. The objectives
of processes and arrangements may be specified explicitly in legislation or be
implicit in the approaches taken to implementation.

2. The analysis of the effectiveness of the different aspects of the system focuses on
the extent to which they achieve their explicit and implicit objectives in practice.

3. The analysis of the efficiency of the different aspects of the system considers the
relationship between the resources committed to the processes and bodies and
results and impact achieved.

These are the core criteria used in many policy and programme evaluation frameworks used
by national and international bodies, including UNESCO and the OECD (UNESCO,
2007,13;; OECD/DAC, 2018;14}). The assessment of effectiveness is often complemented by
consideration of the wider impact of policies and programmes, in an attempt to capture
effects beyond the immediate outputs of the policy or programme in question. A full
analysis of the results and impacts of the Brazilian external quality assurance system would
be methodologically challenging in the absence of obvious counter-factual situations
(higher education not subject to the quality procedures being examined) and accessible,
objective data on quality. In any case, such an impact analysis is beyond the scope of this
review. Nevertheless, the review team has sought to consider the wider effects of the quality
assurance system, to the extent possible.

As noted, the review assesses the relevance, effectiveness and efficiency (and wider
impact) of the higher education quality assurance system in Brazil using two main types of
judgement criteria:

1. Domestic criteria, which are based on objectives, viewpoints and evidence
encountered in Brazil, including the explicit objectives of the Brazilian legislation
and stakeholder perceptions about effectiveness and efficiency and;

2. International criteria, which take into account international standards and
guidelines for effective external quality assurance in higher education and examples
of effective or promising practice in other jurisdictions.

Many variables can affect the design of quality assurance systems in higher education.
National contexts have a strong impact on how quality assurance systems are configured
and there is no one-size-fits-all institutional model of good practice. However, the analyses
of various international associations working in the field of quality assurance in higher
education point to a growing international consensus around a set of principles that can
guide the design of effective external quality assurance. Based on existing international
guidelines and available literature, some of the key attributes one might expect to see in an
effective external quality assurance system, and their relationship to relevance, effective
and efficiency, are summarised in Table 2.2 below.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 61

Table 2.2. Factors in effective quality assurance systems

Aspect of the system Characteristics of effective QA systems
RELEVANCE The objectives of each step in external and internal quality assurance processes are
1. Objectives of QA processes clearly formulated and relevant to the challenges faced by the higher education system.
and the definition of quality An appropriate balance is struck between aiming to eliminate or avoid poor quality
provision and improve existing provision, including provision that is already judged to be of

good quality.

The quality of teaching and learning is conceived of as fitness for purpose, where the

primary purpose of the educational experience is to equip students with relevant learning

outcomes (a student-centred conception of quality).

The definition of relevant learning outcomes takes into account an appropriate range of

knowledge and skills and the level (degree of complexity) of the knowledge and skills

acquired (i.e. transversal skills as well as specific content knowledge).

Definitions of quality are flexible enough to recognise quality in different forms in different

types of educational programme.

EFFECTIVENESS Quality is measured using an appropriately wide range of relevant and reliable indicators,
2. Measurement of quality including input, process and output indicators.
Where appropriate, differentiated indicators are used for different types of provision to

take account of their specific characteristics (e.g. academic vs professional courses).

EFFECTIVENESS Teaching staff and higher education providers are clearly identified as those with primary
3. Responsibility for quality and responsibility for delivering quality.
for quality assurance Subsidiarity: decisions about quality are taken at the lowest level possible while

maintaining effectiveness and adequate accountability.

Quality assurance agency/ agencies act in the public interest, are adequately resourced
and are sufficiently independent from both the higher education sector and government.
Additional government initiatives to promote quality are coordinated with quality assurance
systems to ensure consistency.

EFFECTIVENESS Where appropriate, evidence of poor quality is used to eliminate poor quality provision,
4. Use of information about with demonstrable results.
quality Elsewhere, information about the quality performance of programmes and institutions is

used systematically to improve quality, using an appropriate set of tools, with
demonstrable results.

Information about the quality performance of programmes and institutions is made public
to ensure transparency, with demonstrable results.

EFFECTIVENESS The quality assurance system is able to adapt rapidly and flexibly to take account of
5. Adapting to change and changes in the way teaching and learning are offered or could be offered.
innovation The quality assurance system actively promotes adoption of valuable new course content,

new technologies or learning approaches.
EFFICIENCY The system is cost effective for taxpayers and the system overall and reduces the
6. Cost effectiveness and administrative burden on the education providers to a minimum.

administrative burden

Source: Developed by the OECD Education and Skills Directorate, drawing on INQAAHE Guidelines of Good
Practices 2016 (INQAAHE, 201 6y15]); Standards and Guidelines for Quality Assurance in the European Higher
Education Area (ESG, 2015ys]) and; CIQG International Quality Principles: Toward a Shared Understanding of
Quality (CHEA, 201 6)9)).

Key questions will be addressed for each component of the quality assurance
System

The domestic and international judgement criteria have been used to inform the analysis
and conclusions in the subsequent sections of this report. For each component of the
Brazilian external quality assurance system for higher education (grouped above), the
report addresses the following key questions:

1. Relevance: Are the objectives of this component of the quality assurance system
clear and relevant to needs in the Brazilian context?

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

62 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

2. Effectiveness: Does this component of the system use appropriate indicators
(measures) of quality that would allow it to measure quality in line with the
Brazilian legislation and international good practice?

3. Effectiveness: Is responsibility for assuring the quality of higher education
appropriately distributed between higher education providers and external
authorities and between different external authorities in this component of the QA
system, taking into account the objective of Brazilian legislation and international
experience?

4. Effectiveness: Is the information about quality collected / assembled in this
component of the system used well to inform effective decisions about quality,
promote quality improvement or increase transparency, in line with Brazilian
legislation and international good practice?

5. Effectiveness: Is this component of the system able to adapt flexibility to
accommodate change and does it promote innovation?

6. Efficiency: Is this component of the system efficient in line of the resources
committed and the effects generated. Is it cost effective and does it maintain
administrative burden at a minimum?

Notes

' Undergraduate education (graduagdo) in Brazil encompasses a) 4-6-year bachelor's degrees
(Bacharel), which are academically oriented and account for the majority of enrolment; b)
Licentiate's degrees (Licenciado), which are 3-4-year teacher training qualifications and; c)
professionally oriented, 2-3-year Advanced Technology Programmes (Cursos Superiores de
Tecnologia). All of these qualifications are classified as ISCED 6.

> Public HEIs have de facto accredited status through their acts of establishment, but formally require
periodic re-accreditation.

3 Postgraduate education in Brazil encompasses a) academic master’s degrees; b) Professional
Master’s degrees; and c) doctoral degrees. These qualifications are classified as stricto sensu
postgraduate degrees and classified respectively as ISCED 7 (academic and Professional Master’s)
and ISCED 8 (doctoral degrees). In addition, many higher education institutions in Brazil offer
professionally oriented postgraduate qualifications referred to as Specialisation degrees (Cursos de
especializagao em nivel de pos-graduacao). These are classified as Jato sensu qualifications, which
do not give access to doctoral degrees and are not evaluated by CAPES.

4 The terms “output” and “outcome” are not used consistently in literature about education systems
and educational performance. In this report, qualified graduates with additional knowledge and skills
are classed as an output of the educational process. Outcome is used primarily to refer to the
subsequent labour market outcomes of graduates.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK | 63

References

Brunner, J. and D. Miranda (2016), Educacion Superior en Iberoamérica - Informe 2016 -
Aseguramiento de la Calidad, Centro Interuniversitario de Desarrollo (CINDA), Santiago de
Chile, http://www.cinda.cl (accessed on 13 November 2018).

CAE (2018), CLA+ for Higher Education, Council for Aid to Education,
https://cae.org/flagship-assessments-cla-cwra/cla/ (accessed on 13 November 2018).

CENEVAL (2018), Examenes Generales para el Egreso de Licenciatura (EGEL), CENEVAL,
http://www.ceneval.edu.mx/examenes-generales-de-egreso (accessed on
13 November 2018).

CHEA (2016), The CIOG International Quality Principles: Toward a Shared Understanding of
Quality, Council for Higher Education Accreditation , Washington, DC, http://www.chea..

ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher
Education Area (ESG).

ICFES (2018), Informacion general del examen Saber Pro, Instituto Colombiano para la
Evaluacion de la Educacion, http://www2.icfes.gov.co/estudiantes-y-padres/saber-pro-
estudiantes/informacion-general-del-examen (accessed on 13 November 2018).

INQAAHE (2016), INOAAHE Guidelines of Good Practices 2016 - revised edition,
International Network for Quality Assurance Agencies in Higher Education (INQAAHE),
Barcelona, http://www.ingaahe.org/.

MEC (2017), Decreto desburocratiza e premia instituigdes pela qualidade (Decree reduces
bureaucracy and rewards institutions for quality), Ministerio da Educacao,
http://portal.mec.gov.br/ultimas-noticias/2 12-educacao-superior-1690610854/5861 1-decreto-
desburocratiza-e-premia-instituicoes-pela-qualidade (accessed on 12 November 2018).

OECD (2008), OECD Reviews of Tertiary Education: Mexico 2008, OECD Reviews of Tertiary
Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264039247-en.

OECD/DAC (2018), DAC Criteria for Evaluating Development Assistance, OECD
Development Assistance Committee (DAC),
http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
(accessed on 14 November 2018).

Presidéncia da Republica (2017), Decreto N° 9.235, de 15 de dezembro de 2017 - Dispde sobre
o exercicio das funcgdes de regulagdao, supervisdo e avaliagao das instituigdes de educagao
superior e dos cursos superiores de graduagdao e de pos-graduagao no sistema federal de
ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of
regulation, supervision and evaluation of higher education institutions and undergraduate
and postgraduate courses in the federal education system),
http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on
10 November 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

[7]

[10]

[11]

[9]

[8]

[12]

[15]

[5]

[6]

[14]

[2]

64 | CHAPTER 2. SCOPE OF THE REVIEW AND ANALYTICAL FRAMEWORK

Presidéncia da Republica (2006), Decreto n° 5773 de 9 de maio de 2006, Dispde sobre o
exercicio das funcoes de regulacdao, supervisdao e avaliagao de instituigdes de educacgao
superior e cursos superiores de graduagdo e seqitienciais no sistema federal de ensino.
(Decree 5773 concerning the exercise of the functions of regulation, supervision and
evaluation of higher education institutions and undergraduate and short-cycle programmes
in the federal education system), http://www.planalto.gov.br/ccivil_03/ Ato2004-
2006/2006/Decreto/D5773.htm (accessed on 12 November 2018).

Presidéncia da Republica (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional
de Avaliagdo da Educagao Superior - SINAES e da outras providéncias (Law 10 861
establishing the National System for the Evaluation of Higher Education - SINAES and other
measures), http://www.planalto.gov.br/ccivil 03/_ato2004-2006/2004/lei/110.861.htm
(accessed on 10 November 2018).

Presidéncia da Republica (1996), Lei N° 9.394, de 20 de dezembro de 1996, Estabelece as
diretrizes e bases da educagao nacional. (Law 9394 establishing basic guidelines for
national education), http://www.planalto.gov.br/ccivil_03/LEIS/L9394.htm (accessed on
13 November 2018).

UNESCO (2007), Evaluation Handbook, UNESCO Internal Oversight Service Evaluation
Section, Paris, http://unesdoc.unesco.org/images/0015/001557/155748E.pdf (accessed on
13 November 2018).

[4]

[1]

[3]

[13]

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 65

3. Higher education in Brazil

This chapter presents a brief overview of the higher education system in Brazil and the
underlying socio-economic context in which it operates. It starts with a short review of
recent macroeconomic developments and demographic trends, before examining the main
legal and administrative governance arrangements for the public and private higher
education institutions that make up the federal higher education system and account for
the vast majority of institutions and student enrolment in Brazil. The chapter then presents
key data relating to the institutional landscape in higher education, types of programme
offered, the teaching workforce, enrolment and attainment rates, a discussion of social
equity and evidence about the learning and employment outcomes for graduates.

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The
use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli
settlements in the West Bank under the terms of international law.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

66 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

3.1. Focus of this chapter

Over the past decade, Brazil has seen rapid growth in participation in higher education,
mostly enabled through the expansion of private higher education provision (de Magalhaes
Castro, 2015;1;). Enrolment in higher education increased from less than six million
students in 2009 to over eight million in 2016, with over 75% of students now studying in
private institutions (MEC, 2018 (unpublished),2;). In the decade up to 2017, the tertiary
education attainment rate among young adults in Brazil (aged 25-34) increased from 10%
to 17%. The average rate of tertiary education attainment for this age group in OECD
countries is 43% (OECD, 20183).

The increase in higher education attainment in Brazil mirrors trends seen in other OECD
and partner countries in the last decade, albeit from a lower starting base and with current
attainment rates remaining low by international standards. Further expansion of higher
education enrolment is an explicit objective of Brazil’s current National Education Plan,
which aims for a third of 18-24 year-olds to be enrolled in higher education by 2024 (MEC,
2014;4)).

This objective is consistent with policies pursued by governments across the world.
Increasing levels of higher education attainment have long been associated with economic
competitiveness in an increasingly knowledge-driven global economy (OECD, 2018;5)). As
Brazil’s economy and labour market expand in more knowledge-intensive sectors, demand
for higher-level skills is likely to increase.

However, with expansion, increased concerns about the quality of higher education in
Brazil have emerged, particularly in some sections of the system (Salto, 2018;6}). Against
this backdrop, Brazilian authorities have also focused considerable efforts on regulating
private higher education providers and implementing mechanisms to assure the quality of
higher education provision in all parts of the system. This report reviews the systems in
place in Brazil to assure the quality of higher education. To provide context for the
discussions of quality assurance that follow, this chapter provides a brief overview of the
broad socio-economic context in which Brazil’s higher education system operates as well
as key characteristics of the system itself.

3.2. The socio-economic context for higher education in Brazil
Economic conditions

Brazil’s economy is gradually emerging from the recession

In the early 2000s, macroeconomic stability, positive demographic trends and favourable
global economic conditions, including rising commodity prices, allowed the Brazilian
economy to grow, leading to high employment rates, wage growth and an expansion of
private and public consumption. The strong economic context, coupled with improving
access to school education and extensive government transfer programmes have allowed
an estimated 25 million Brazilians to escape poverty since 2003 (OECD, 2018)s;).

However, this model of economic growth reached its limits by the middle of the current
decade. A rapidly-ageing population, deteriorating trade performance, political instability
and rising public debt led to a deep and prolonged recession in 2015 that wiped out almost
seven years of growth and doubled unemployment. Poverty levels have stagnated and
Brazil remains one of the most unequal countries in the world. Nevertheless, since 2017,

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 67

growth has resumed and annual inflation and unemployment have started to decline
(Figure 3.1.)

Figure 3.1. GDP growth and unemployment in Brazil

Gls GDP growth, year-on-year (left axis) Unemployment rate (right axis)

% %
6 14
4 12
2 10
0 8
-2 6
-4 4
6 2
8 0

\ \ Ss v 7 v Gy e) ~) & \& \& NC) ) NS) © © © \ \
NY YN NM GM HM SC SB KH KS HB KB FE LC LS LF LC LC AX YS
ve ys we ve wy Ro ve ys RS ve \S we we ~S RS we ys we

Source: (OECD, 2018j5]) OECD Economic Surveys: Brazil 2018, OECD Publishing Paris,
http://dx.doi.org/10.1787/eco_surveys-bra-2018-en.

Stronger investment and productivity are key for future growth

In this context, the OECD’s most recent economic survey of Brazil (2018;s}) argues that
stimulating growth and social progress will require strong investment across the economy.
The OECD economists call for policies to reduce administrative burdens, simplify taxes
and streamline licensing of economic activities. Raising productivity, which has been
largely stagnant over the last 15 years, will also be crucial for future economic
development. Improvements in productivity require not only more investment in physical
capital, but also in the skills of people (OECD, 2018)5)). Ensuring access to high-quality
higher education for all will be key for productivity growth.

Demographic trends and social conditions

Brazil has a young population, that is beginning to age

At 25%, the share of young people (aged 15-24) in the working-age population in Brazil is
currently high in comparison to most OECD countries. However, by 2050, the proportion
of young people in the working-age population in Brazil is forecast to have fallen to below
the average of OECD and selected emerging economies (OECD, 2014;7)). This will mean
fewer young people are available to generate wealth to help support the rest of society and
each individual will need to be more productive just to maintain current living standards.
Studies suggest that a highly educated - and thus more productive - population will be key
to responding to the challenges of a higher old-age dependency ratio (Dwyer et al., 2016ys)).

High levels of inequality

In contrast to many OECD countries, Brazil has seen a decrease in income inequality over
the past decade. However, the Gini coefficient - measuring income inequality - remains

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

68 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

higher than in any OECD country (Figure 3.2). Half of the population receives 10% of total
household income, while the other half holds 90% (OECD, 2018)s)).

Figure 3.2. Income inequality

Gini coefficient, 0 = complete equality; 1 = complete inequality, 2017 or latest available

0.7

0.6

0.5

0.4

0.3

0.2

0.1
SEE ECI SELF EEK KEG HEC LILLE PEO
Cor eew TP PGP LPH CG FS oss VRQ OF LNT Se

x gs ao & s os

Source: (OECD Data, 2018j9}), Income inequality, https://data.oecd.org/inequality/income-inequality.htm.

Significant inequalities also exist along geographical and ethnic lines. For example, a 2014
study calculated that the likelihood of a young, black woman living in the Northeast of
Brazil being unemployed was 28.6%. This compared to an unemployment rate for white
males living in the South of 7.6% (OECD, 2014;7)). Among the richest 1% of the
population, less than one in five of are black or mixed race, even though these groups
account for over 50% of the total population. Among the poorest 10% of the population,
over 70% are black or mixed race (IBGE, 2014110)).

Crime and corruption are widespread in Brazil

Brazil ranks 96" out of 180 countries in the latest Transparency International corruption
index (Transparency International, 2018;11;). Scandals that surfaced in relation to
corruption in public procurement, including by state-owned companies, and infrastructure
concessions in recent years have created significant political turmoil. The OECD (2018;s})
has highlighted improving transparency and accountability as a key priority to tackle the
root causes of corruption.

According to the latest OECD data, Brazil's homicide rate is 27.6 per 100 000 population,
more than seven times the OECD average of 3.6 (OECD, 2017;12)). In addition to being a
social and criminal justice issue, reports suggest that high levels of violence have a negative
impact on economic growth due to the direct costs of crime and as an indirect constraint
for business growth (World Bank, 2006113).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 69

3.3. Governance arrangements, funding and key public policies for higher education
Governance of higher education

Responsibility for higher education is shared between the Union, states and
municipalities

Higher education provision in Brazil is a shared responsibility between the federal
government, the 27 federative units (the 26 states and the federal district of Brasilia) and
the municipalities. The federal authorities, state governments and municipalities are all
permitted to create and fund the operation of public higher education institutions. In
practice, the involvement of state and municipal authorities in providing higher education
varies considerably between states across the country. The federal government, through the
National Education Plan, establishes the national strategy for higher education. It has also
assumed primary responsibility for funding student aid programmes, is responsible for the
external quality assurance of federal public higher education providers and has exclusive
responsibility for licensing and assuring the quality of private higher education providers
throughout the country. State governments are responsible for regulating and assuring the
quality of state and municipal public institutions in their territories.

Ministry of Education steers, regulates and supervises the federal higher education
System

The federal Ministry of Education (Ministério da Educagado, MEC) is responsible for
establishing national education policy at the federal level, as well as coordinating the
different levels and parts of the education system. The federal higher education system
comprises both federal public higher education institutions and all private higher education
institutions in Brazil.

In higher education, MEC is currently responsible for establishing, funding, steering and
regulating the federal Higher Education system, through its different secretariats
(Figure 3.3), which, at the time of writing, are:

e The Secretariat for Higher Education (Secretaria de Educagao Superior, SESU)
establishes, funds, and steers the network of federal universities and is, de facto,
responsible for developing the overall strategy of the federal government in higher
education.

e The Secretariat for Vocational and Technological Education (Secretaria de
Educagdo Professional e Tecnologica, SETEC) coordinates, monitors and
evaluates vocational education and training in the country, including the (relatively
small) network of federal institutes of education, science and technology that
provide Advanced Technology Programmes (cursos superiores de tecnologia)
alongside non-tertiary professional programmes.

e The Secretariat for Regulation and Supervision of Higher Education (Secretaria de
Regulagdo e Supervisdo da Educagao Superior, SERES) supervises and regulates
the system, including the approval of new institutions and programmes. SERES is
the main ministry department involved in the Brazilian quality assurance system
for undergraduate education and some types of professionally oriented (/ato sensu)
postgraduate education (the Sistema Nacional de Avaliagao da Educagao Superior,
SINAES).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

70 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

National level

Figure 3.3. Current governance arrangements for the federal higher education
system in Brazil (November 2018)

National Council for

Education (CNE) Minister of
Education

National Commission (M EC) .
for Evaluation in |
Higher Education ,
(CONAES) |
|
I
Executive Minister’s 7
Secretary Cabinet |
|
| |
|
Unit for Unit for Unit for Unit for Unit for .
Vocational Basic Regulation Higher Education |
Education education and Education System Co- |
and (SEB) Supervision (SESU) ordination :
Training of Higher (SASE) |
(SETEC) Education 7
(SERES) |
|

Indirect administration

professional education education centres
institutes (IFEPT) (CEFET)

Note: This figure does not provide a complete overview of the education system in Brazil, nor does it include
the different state and municipal level governance systems.

Source: (MEC, 2018,14]), Estrutura Organizacional - Ministério da Educagdo (Organisational Structure -
Ministry of Education), http://portal.mec.gov.br/institucional/estrutura-organizacional.

MEC is supported in the regulation, evaluation and supervision of higher education by two
main advisory bodies and two specialised implementation agencies with varying degrees
of autonomy:

The National Council for Education (Conselho Nacional de Educagdo, CNE) provides
advice for the development and assessment of national educational policy, including the
National Plan for Education (Plano Nacional de Educacgdo, PNE). The CNE is composed
of 24 members nominated by the President for a four-year mandate. The CNE is composed
of two chambers: one for basic education (Camara de Educac¢ao Basica) and one for higher
education (Camara de Educagdo Superior).

The National Commission for Evaluation of Higher Education (Comissdo Nacional de
Avaliagao da Educagao Superior, CONAES), established in 2004, is an advisory body
responsible for overseeing the implementation and further development of the national
system of evaluation of higher education, SINAES. CONAES is tasked with assessing the
mechanisms of institutional, programme and student evaluation, developing proposals for
the development of higher education institutions, defining curriculum parameters for each
field, articulating with the state education systems, among others.

Federal universities Technological and Federal technological

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 71

The Anisio Teixeira National Institute for Educational Studies and Research (nstituto
Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, INEP), established in
1937, is a semi-autonomous agency responsible for implementing SINAES. It coordinates
the collection of data and is tasked with developing and implementing key elements of
quality assessment in higher education, including the National Examination of Student
Performance (Exame Nacional de Desempenho de Estudantes, ENADE), programme
evaluation and institutional evaluation.

The Foundation for the Coordination of Improvement of Higher Education Personnel
(Coordenagao de Aperfeigoamento de Pessoal de Nivel Superior, CAPES), established in
1951, is a public foundation under the responsibility of MEC. CAPES has been responsible
for assuring the quality of academic (stricto sensu) postgraduate programmes since the mid-
1970s. CAPES operates an extensive system of accreditation and quality rating of
postgraduate provision based primarily on peer review. In addition, the Foundation
provides public funding to research and promotes international scientific cooperation.

The Ministry of Science, Technology, Innovation and Communication (Ministério de
Ciéncia, Tecnologia, Inovagées e Comunicagées, MCTIC) provides funding for research
and innovation in higher education institutions, primarily through its associated executive
agency, the National Council for Research and Development (Conselho Nacional de
Desenvolvimento Cientifico e Tecnologico, CNPq).

Funding higher education

A hybrid higher education system, with significant private provision

The Brazilian higher education sector is classified as a “hybrid system” (Ferreyra et al.,
2017{15}), with significant public and private higher education sectors and differences
between these in terms of funding. As established in the 1988 Constitution, public
institutions may not charge tuition fees to students and rely almost exclusively on public
funds for their operation. As in other countries, there are opportunities for public
institutions to obtain resources from other funding streams (Corbucci and Fonseca
Marques, 2003/16), through donations and by providing services. However, consolidated
recent data on these funding streams are not available for public institutions in Brazil.

Institutions in the private sector receive no direct institutional subsidies and depend on
income from student fees. Private institutions are free to determine the level of fees charged
(Dwyer et al., 2016;8)). In 2017, students in the private sector paid an average monthly fee
of BRL 898 (EUR 281) (SEMESP, 2017{17)), although this figure masks considerable
variation between institutions and programmes.

Private institutions may benefit from indirect public subsidies, through public grant and
loan programmes provided to low-income students attending private institutions (FIES and
PROUNI - see below). Students in public institutions do not have access to publicly funded
maintenance grant and loan schemes. Private not-for-profit institutions may also benefit
from indirect tax and social security exemptions, which were estimated to amount to
BRL 9 billion in 2013 (Davies, 2017118).

Public spending on higher education is slightly below the OECD average

Total public expenditure in Brazil on educational institutions from primary to tertiary level
represents 5% of the country’s GDP. This comparatively high level of public expenditure
mostly reflects comparatively high spending on primary to post-secondary non-tertiary

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

72 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

education (4% of GDP) (OECD, 2018;3}). However, these data do not take into account
private and public spending on student support schemes (as opposed to institutional
subsidies), which through assisting students to pay fees, form an indirect subsidy to private
higher education institutions. If this expenditure were accounted for, studies suggest that
expenditure on tertiary education institutions in Brazil as a percentage of GDP would be
higher than the average in OECD countries (Nascimento and Verhine, 2017119).

Figure 3.4. Total expenditure on educational institutions as a percentage of GDP (2015)

From public, private and international sources, % of GDP.

B® Primary, secondary and post-secondary non-tertiary Tertiary

” OPP GLP EHP RLS PLP I LP LLP LL LE CEL PB PSSM S

@".: 2)
ve s £ SS ge ee

Note: Data for Brazil take into account public expenditure to public institutions only (federal, state and
municipal-level institutions). Private spending and public spending on student support schemes (as opposed to
institutional subsidies), which through assisting students to pay fees, form an indirect subsidy to private higher
education institutions are not accounted for.

Source: (OECD, 2018)3)), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2018-en.

In 2015, public subsidy per student to public tertiary education institutions in Brazil,
including research and development (R&D) activities and adjusted for purchasing power
parity (PPP), was USD 14 261. This is close to the level of public subsidy to public
institutions in OECD countries such as France (USD 14 386) and the Netherlands (USD 14
369). These data refer exclusively to public spending per student in public institutions, as
private institutions do not receive direct public subsidies and data on private spending on
public higher education in Brazil are not available. The figures thus reflect spending levels
that affect less than 25% of Brazilian students.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 73

Figure 3.5. General government and private expenditure per full-time tertiary student in
public higher education institutions, (ISCED 2011 levels 5 to 8), including research and
development (2015)

In equivalent USD converted using PPPs for GDP, by level of education, based on full-time equivalents

25000
20000
45000
40000
5000 al
0 1
se

@ General government OPrivate

- R
s

F S & & & « ¥. we > “8 Se
a s & rr se & wae se Ss & 3s & w ee RS
e w ? 7 “s "s oe # og g ss
NSC &

Note: * Data for the United Kingdom refer to institutions that formally have private, not-for-profit, legal status,
but which have historically been government-dependent and are considered to be public institutions in national
policy documents. Data on private expenditure on public higher education institutions are not available for
Brazil, although in the absence of tuition fees, private revenues are likely to be low.

Source: (OECD, 2018,3]), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2018-en.

Expenditure per tertiary student declined at the beginning of the current decade and in 2014
was equivalent to 80% of the total value for 2010. In comparison, expenditure per student
at primary, secondary and post-secondary non-tertiary level had increased by 58 percentage
points since 2010 (OECD, 2017;20)).

Public spending in tertiary education is comparatively centralised

While 92% of final funds for pre-primary and school education (after transfers between
levels of government) are managed by regional and local governments in Brazil, around
80% of final public funds for higher education are managed by the federal government.
The remaining 20% is managed primarily by state governments (OECD, 2018)3)). The
significance of the state and municipal public higher education sectors varies considerably
between states in Brazil.

The federal Ministry of Education allocates operating budgets to federal higher education
institutions on an annual basis, based on historical patterns for current expenditure and on
funding formulas for capital expenditure. While federal institutions have a certain degree
of autonomy in resource allocation, the Ministry allocates a specific share for current
expenditure and another for capital expenditure (Corbucci and Fonseca Marques, 2003/16))

State-level universities receive funding from the state government. In some cases, the
resource allocation for each university is tied to the state-level budget. In others, such as
the State of Sao Paulo, universities receive a specific share of state-level taxes. State-level
institutions tend to have greater autonomy to allocate such resources internally.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

74 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

Funding higher education is a politicised topic

There is growing controversy regarding the way higher education is funded in Brazil, as
students from more advantaged backgrounds are more likely to access highly selective elite
public institutions, which are free, whereas those from poorer backgrounds tend to attend
fee-paying private institutions (McCowan, 2007/21;). This is widely seen to exacerbate
socio-economic inequalities (OECD, 2014;7;). In 2017, the Brazilian Congress blocked a
proposal that would have allowed public higher education institutions to charge tuition fees
for specialisation and professional (/ato sensu) postgraduate programmes (Portal da
Camara dos Deputados, 2017;22)).

In 2016, the Brazilian government approved an expenditure rule that sets a ceiling on
federal expenditure for the next 20 years to stabilise public debt. In practice this will freeze
future primary expenditure at 2016 levels, adjusting to inflation (OECD, 2018;5)).
Previously the Constitution defined a minimum percentage of revenue to the education and
health sectors - 18% of net tax revenue and 15% of net current revenue, respectively. The
new rule also protects expenditure to these sectors, but alters the mechanism for doing so,
by defining spending floors for education and health, whereby expenditure for these sectors
should be no lower than their 2017 expenditure ceiling (IMF, 2017,23)).

Key federal government policies for higher education

Strategy focused on expansion and quality

In recent years, the federal government policy has focused on increasing access to higher
education, in particular for disadvantaged socio-economic groups, as well as promoting
improvements in quality. In 2014, the federal government adopted the National Plan for
Education (PNE) for 2014-24, which sets 20 goals for improving access to education and
quality from early childhood to adult education. Three of the goals relate explicitly to higher
education (Box 3.1). INEP is tasked with monitoring implementation of the PNE and
assessing results at the federal level. State and municipal level governments are responsible
for the implementation and monitoring PNE in their jurisdictions.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 75

Box 3.1. Higher education goals in the National Education Plan 2014-2024

Target 12: by 2024, raise gross enrolment in higher education to 50% and net enrolment!
to 33% of the 18-24 year-old population, ensuring the quality of provision and with at least
40% of new students enrolling in public institutions. Related measures include:
infrastructure improvement, increasing the number of study places, raising completion
rates, offering at least a third of classes in the evening, expanding student support systems,
and adopting affirmative policies. In 2017, the gross enrolment rate was 35% and the net
rate was 23%.

Target 13: by 2024, raise the quality of higher education and increase the qualification
level of the entire teaching workforce in public and private institutions. Ensure that at least
75% of teaching staff have attained a bac’s degree and 35% a doctoral degree. Relevant
measures include: improving the National System of Higher Education Evaluation
(SINAES), encouraging institutional self-assessment, expanding the National Examination
of Student Performance (ENADE) and ensuring that by 2024, three-quarters of students
correctly answer at least 75% of the items on the exam. In 2017, 78% of teaching staff had
master’s degrees and 40% a PhD.

Target 14: by 2024, increase participation in stricto sensu graduate programmes. The goal
is to award 60 000 master’s and 25 000 doctoral degrees per year. Relevant measures
include: expanding financial support to stricto sensu graduate programmes, articulating
CAPES and state-level R&D support agencies and expanding the offer of stricto sensu
graduate programmes. In 2016, 59 600 stricto sensu master’s degrees were awarded and

20 600 PhDs.
Sources: (Presidéncia da Republica, 2014,[24}) Lei No.13005 - Aprova o Plano Nacional de Educagado (PNE)
(Law No. 13005 - Approval of the National Education Plan),

http://www.planalto.gov.br/CCIVIL _03/ Ato2011-2014/2014/Lei/L13005.htm, (INEP, 2018,25]) Relatorio do
2° ciclo de monitoramento das metas do plano nacional de educagao (Report on the second cycle of monitoring
of the goals of the National Education Plan).

Modest expansion of public institutions and student support programmes for
students in the private sector

Over the last two decade, federal higher education policy has been characterised by
investment in expanding and improving federal higher education institutions and student
aid programmes aimed at helping students from low-income backgrounds to access courses
in the private sector.

The Programme for Restructuring and Expansion of Federal Universities (Programa de
Apoio a Planos de Reestruturagado e Expansdo das Universidades Federais, REUNI),
launched in 2007, provided funds to create additional study places and increase
completion rates in federal institutions (Presidéncia da Republica, 2007)26)). In parallel,
the “University for All” Programme (Programa Universidade para Todos, ProUni),
created in 2005, provides tax exemptions to private institutions that offer free or reduced
tuition fees to students from low-income families or who attended public upper secondary
education. The Higher Education Student Funding Programme (Financiamento Estudantil
no Ensino Superior, FIES), created in 1999, is a public student loan system which offers
subsidised interest rates and comparatively generous repayment terms to students at
private institutions. A reduction in interest rates and an extended reimbursement

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

76 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

timeline introduced in 2011 led to a significant increase in demand for loans - from
34 654 in 2009 to more than 700 000 in 2014 (Salto, 2018j6}). In total, FIES and ProUni
covered approximately 22 percent of all private HEI students in 2014, of whom 50%
identified as black (Zalaf Caseiro, 2016;27)).

However, a report by the Federal Court of Accounts (Tribunal de Contas da Unido, TCU)
showed that FIES’ expansion was conducted without adequate planning and that did not
lead to a significant expansion of net enrolment. Given high default rates and the 2015
economic recession, the Brazilian government has since established a cap of 250 000
loans/year and stricter conditions (TCU, 2016)8)).

Quotas in federal universities aim to mitigate social inequities

With the aim of tackling social inequities in the student body in public institutions, the
Brazilian government introduced the Quota Law (Lei das Cotas), approved in 2012 - an
affirmative action law which imposes quotas on federal universities for the recruitment of
students from disadvantaged backgrounds. By 2016, 50% of enrolments were reserved for
students from public secondary schools (25% for students with a per capita family income
below one and a half minimum wages). Institutions are also be expected to respect
minimum proportions of ethnic minority students, based on census statistics for the region
where they are located (OECD, 2014;7)).

There have been moves to standardise admissions procedures

In 2010, the federal authorities introduced the Unified Selection System (Sistema Unificado
de Selegado, SISU), with the objective of streamlining access processes for higher education.
In the past, HEIs offered their own entrance examination (known as the vestibular) and the
National Examination of Upper Secondary Education (Exame Nacional do Ensino Medio,
ENEM) acted as a secondary school-leaving examination. Improvements to the ENEM
content and administration, as well as a decision by the Ministry to use the ENEM as the
only admission criteria for entry into federal universities (2009) have led to an increasing
proportion of public and private HEIs using ENEM results to select students at entry.
Institutions retain the autonomy to choose how to use students’ ENEM results. They might
be the exclusive criteria or used as a complement to individual entry examinations.
Programmes may also determine their own minimum grade requirements for overall
ENEM results or for specific subjects (MEC, 201729).

3.4. Higher education provision in Brazil
A diversified institutional landscape

A legal distinction between types of higher education institution

Public and private higher education institutions (HEIs) in Brazil are formally classified into
three categories:

Colleges (faculdades): smaller, teaching institutions often dedicated to a specific field.
Colleges currently account for 83% of HEIs in the country.

University centres (centros universitarios): comprehensive institutions, mainly dedicated
to teaching. University centres offer some postgraduate programmes and may conduct
research, but there is no requirement to do so. University centres have greater autonomy in
creating new programmes than colleges.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 77

Universities (universidades): comprehensive institutions that are expected to conduct
research and to offer postgraduate education. Universities also have autonomy to create
new programmes.

Additionally, there are 38 Federal Institutes for Education, Science and Technology
(Instituto Federal de Educagdo, Ciéncia e Tecnologia) and two Federal Technological
Education Centres (Centros Federais de Educagao Tecnologica, CEFETs), which are
federal public institutions.

A higher education landscape dominated by private providers

The vast majority of new higher education institutions created in the last two decades have
been private. In 2016, 87% of Brazil’s 2 407 HEIs were private institutions (MEC, 2018
(unpublished),2}) the majority of which (88%) were categorised as colleges. These private
institutions enrolled over 75% of undergraduate students. For-profit providers - legally
authorised in 1988 - play an increasingly important role and are often controlled by large
business groups, such as Kroton and Anhanguera, which are publicly traded (de Magalhaes
Castro, 2015;1;). In 2016, for-profit institutions represented 44% of the total number of
institutions and 42% of students enrolled at ungraduated level.

Among the 296 public institutions, there are 107 federal, 123 state and 66 municipal
institutions that enrol 15.5%, 7.7% and 1.4% of its students respectively (MEC, 2018
(unpublished);2)). State and municipal institutions are mostly concentrated in the Southeast
and Northeast regions of the country, whereas federal institutions are more evenly spread
out across the country (MEC, 2018 (unpublished),)).

Table 3.1. Number of tertiary education institutions, by type and sector (2016)

Type
TOTAL College University centre University IF & CEFET
Total 2407 2004 166 197 40
Public 296 138 10 108 40
Federal 107 4 - 63 40
State 123 83 1 39 na.
Municipal 66 51 9 6 na.
Private 2111 1 866 156 89 na.
For profit 1052 978 54 20 na.

Not for profit 1059 888 102 69 na.

Note: IF = Federal Institutes for Education, Science and Technology; CEFET = Federal Technological
Education Centres

Source: (MEC, 2018 (unpublished),2]), Censo da Educagdo Superior 2018 (Higher Education Census 2018),
Data supplied by MEC on 5 October 2018.

Universities and university centres have higher levels of autonomy

The 1988 Constitution grants universities and university centres, whether public or private,
higher levels of autonomy than colleges (faculdades), in particular regarding academic and
administrative matters. They have the right to establish new undergraduate and /ato sensu
postgraduate programmes and alter the number of study places in existing programmes
without prior authorisation from MEC? and issue and register diplomas for their own
programmes. Colleges must rely on accredited universities to register their diplomas. In
addition, universities and university centres have the freedom to develop curricula - while

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

78 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

following national curriculum guidelines - establish research programmes, allocate
resources and expand geographic coverage.

However, public higher education institutions are subject to civil service regulations
regarding their teaching workforce. They must follow strict remuneration and hiring
procedures as permanent staff have the status of public employees. In addition, public
institutions are subject to public sector regulations on purchasing and contracts. This has
led many HEIs to establish separate foundations (fundacgées de apoio), subject to private
law, that allow for greater administrative autonomy (Schwartzman, 2003,30)).

Undergraduate and postgraduate programmes

Different types of degrees in undergraduate and postgraduate education

Authorised HEIs may provide all kinds of undergraduate (ISCED 6) and postgraduate
degrees (ISCED 7 and 8):

Bachelor’s degree (bacharelado): four-to-five-year degrees that usually lead to legally
recognised and regulated professions (e.g. law and medicine).

Teacher’s license degree (/icenciatura): four-year degrees that allow graduates to teach in
pre-primary, primary and secondary education.

Advanced Technology Programmes (cursos superiores de tecnologia): three-year
vocational and professionally oriented programmes.

Specialisation (especializagdo): two-year programmes that are more professionally
oriented, such as Master’s in Business Administration (MBAs).

Master’s degrees (mestrados): two-year programmes that may be professionally oriented
or have a stronger academic focus.

Doctorate degrees (doutorados): four-year programmes with a strong academic and
scientific focus.

Seven out of ten undergraduate students are enrolled in a bachelor’s degree (bacharelado).
Nearly 20% of students at the undergraduate level are enrolled in a teacher training degree
(licenciatura) and around 12% in vocational programmes (cursos superiores de
tecnologia).

Table 3.2. Undergraduate enrolment, total and share

Number of students enrolled, by type of .degree and sector. In parenthesis the share of total enrolment (%).

Bachelor's degree Teacher training sito
ene Technology Other programmes Total
(bacharelado) degree (licenciatura) Programmes
Public Federal 823 295 (10.2%) 328 032 (4.1%) 73 951 (0.9%) 24 175 (0.3%) 1 249 453 (15.5%)
State 305 990 (3.8%) 229 781 (2.9%) 79 726 (1%) 8 ni (0.1%) 623 710 (7.7%)
Municipal 92 684 (~0%) 21 301 (0.0%) 3,323 (~0%) 27 (~0%) 117 335 (0.3%)
Private — For profit 2 249 357 (27.9%) 593 227 (7.4%) 543,730 (6.8%) 1015 (~0%) 3 387 329 (42.1%)
Not for 2.078 410 (25.8%) 348 153 (4.3%) 245,499 (3%) 2 365 (~0%) 2 674 427 (33.2%)
profit
Total 5 549 736 (68.9%) 1 520 494 (18.9%) 946 229 (11.8%) 35 795 (0.4%) 8 052 254

Source: (MEC, 2018 (unpublished),2}), Censo da Educagado Superior 2018 (Higher Education Census 2018),
Data supplied by MEC on 5 October 2018.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 79

Growing postgraduate provision, with idiosyncratic programme classification

In contrast to the situation in most OECD countries, in Brazil, master’s-level programmes
(ISCED 7) are divided into two categories: “stricto sensu” and “lato sensu” programmes.
master’s degrees (mestrado académico) and Professional Master’s degrees (mestrado
profissional) are classified as stricto sensu given their strong academic and scientific focus.
More professionally oriented, postgraduate “specialisation” programmes, including MBAs,
are Classified as /ato sensu provision (CAPES, 2018)31)). Lato sensu programmes are not
regulated individually. HEIs are allowed to offer them if they meet certain criteria related
to staff qualifications and programmes, but institutions must inform MEC of their
existence. In 2017, 22.5% of master’s students were enrolled in Jato sensu programmes
(MEC, 2018)32)).

Brazil has witnessed a significant expansion of postgraduate education in the last two
decades. The number of master’s and doctoral programmes increased more than three-fold
between 1996 and 2014 (Figure 3.6). The relative importance of Professional Master’s
programmes has also been increasing since the late 1990s. By 2014, Professional Master’s
represented 14% of all master’s programmes offered and 11.4% of master’s degrees
awarded.

Figure 3.6. Number of stricto sensu master’s and doctoral programmes, 1996-2014

TH Master programmes Doctoral programmes

4000
3500
3000
2500
2000
1500
1000

500

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Source: (CGEE, 2016)33)), Mestres e doutores 2015 - Estudos da demografia da base técnico-cientifica
brasileira (master's and doctoral graduates 2015 - a demographic study of the Brasilian technical and science
base), Centro de Gestdo e Estudos Estratégicos, Brasilia, http://www.cgee.org.br.

The publication of Brazilian science and engineering articles increased on average by
11.8% a year between 2003 and 2013 (OECD, 2016;34}). Brazil’s citation impact increased
from 0.73 in 2011 to 0.86 in 2016. However, only 6.4% of Brazilian papers were in the
world’s top 10% most cited, below China (11%) and South Africa (10.2%) (Clarivate
Analytics, 2018,35))

Distance education has been expanding, particularly in the private sector

In the past decade, participation in distance education has expanded significantly from
4.2% of total enrolment in 2006 to 18.6% in 2016. The share of undergraduate students in
the private sector that are enrolled in distance programmes, at 22%, is larger than in federal

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

80 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

public (5.9%) or state-level institutions (7.3%). Currently 92% of students enrolled in
distance undergraduate degrees are in a private institution (MEC, 2018 (unpublished))2)).

Teaching workforce

According to the 2016 Census of higher education, there were 398 000 higher education
teachers in the country, of whom 55% were in private institutions, 30% in federal
institutions and the remaining 15% in state and municipal HEIs. On average, the ratio of
students per teacher is 17.1 in Brazil, although this varies considerably across institutions
and sectors, ranging from 10.7 in Federal faculdades to 21.8 in private universities (INEP,
2016y36))

The qualifications of teaching staff vary between institutional types

As shown in Figure 3.7, over 50% of teaching staff in private and public universities have
doctoral degrees, compared to 30% in federal technical institutions, 23% in university
centres and 18% in colleges. The prevalence of teachers with doctoral degree is higher in
public HEIs (59%) than in private institutions (22%).

Figure 3.7. Share of professors by educational attainment, by sector

BNo qualification Undergraduate education D Specialisation B Master's degree ODoctorate degree

Public (total) Federal State Municipal Private (total) Total

Source: (INEP, 2016)36}), Sinopses Estatisticas da Educagao Superior - Graduagao - INEP (Sinopsis Higher
Education Statistics - Undergraduate Education - INEP), http://portal.inep.gov.br/web/guest/sinopses-
estatisticas-da-educacao-superior.

On average, across all institutional types, half of teaching staff (52%) are hired on a full-
time basis, another 27% are hired as part-time staff and 21% are hired by the hour. In public
institutions, the share of full-time professors is much larger (85%) than the average in
private HEIs (26%), with the highest proportion for full-time permanent staff in federal
institutions (where staff are civil servants). Data from Brazilian household surveys suggest
the earnings of a median professor in higher education correspond to the 96" earnings
percentile, which means that only four percent of workers earn more (Ferreyra et al.,
2017,15)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 81

Table 3.3. Teaching staff by contract, sector (2016)

Full-time Partial Per hour
Public (total) 85% 11% 3%
Federal 92% T% 0%
State 76% 18% 6%
Municipal 37% 29% 34%
Private (total) 26% 40% 35%
Total 52% 27% 21%

3.5. Main trends in participation, equity and outcomes
Participation and attainment

Increasing rates of enrolment and attainment

In 2017, 17% of 25-34 year-olds in Brazil had a tertiary education qualification, compared
to 10% in 2007. Nevertheless, as noted earlier, tertiary attainment among young adults (25-
34) in Brazil still lags behind the average of OECD countries (43%), and is below all other
Latin American countries with available data: Argentina (18%), Chile (30%), Colombia
(28%), Costa Rica (28%) and Mexico (23%).

Figure 3.8. Educational attainment of 25-34 year-olds (2017)

Percentage of 25-34 year-olds with a given level of education as the highest level attained

B Below upper secondary © Upper secondary or post-secondary non-tertiary Tertiary

100

90
80
70
60
50
40
30
20
10

LM PML AS, FW VF OW _ @ WM
RO Ow Loe procs NS ty AV ALO ONG
SS Sf ©)

NS we RS ry Karey
S x

Source: (OECD, 2018,3)), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2018-en.

The higher levels of educational attainment are a reflection of increased participation in
higher education. The total number of students enrolled in undergraduate programmes in
Brazil has increased nearly four-fold in the last two decades, from 1.7 million students in
1995 to roughly six million in 2009 and over eight million students in 2017 (Figure 3.9).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

82 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

Figure 3.9. Enrolment in undergraduate programmes

HE Public Private

7000000

6000000

5000000

4000000

3000000

2000000

0 1 4 1 4 1 1

2009 2010 2011 2012 2013 2014 2015

Note: Data include enrolment in distance and non-distance education.

Source: (INEP, 2018,37]), Numero de Matriculas da Educagado Superior Por Categoria Administrativa e
Abrangéncia Geogrdfica (Enrolment in Higher Education, by region and administrative category),
http://inep.gov.br/inep-data.

Strong growth in enrolment in the private sector

Participation in the private sector has not only experienced a significant increase in absolute
terms, but also relative to the public sector. In 2017, the private sector represented over
75% of enrolment in undergraduate programmes (INEP, 2018;37})compared to only 58% in
1994. However, the private sector’s weight is not as important in postgraduate education
although the number of programmes in private HEIs has expanded significantly in the past
two decades (CAPES, 2018)31)). Less than one in three postgraduate students is enrolled in
a private institution and this share is even lower for those attending more academically
oriented (stricto sensu) programmes, in particular master’s (15.4%) and doctoral degrees
(11.5%).

Distance education has expanded, driven by the private sector

In absolute terms, enrolment in distance education programmes doubled between 2009 and
2017 and now accounts for over 1.7 million undergraduate students (INEP, 2018;38)).
However, a more fine-grained analysis shows that expansion has occurred exclusively in
the private sector. Since 2012, there has been a decline in the number of undergraduate
students enrolled in public distance education programmes (Figure 3.10). Nearly seven in
ten students enrolled in distance programmes were attending for-profit institutions in 2014
(Salto, 20186). Tuition fees in distance education programmes are on average considerably
lower than in classroom-based (presencial) programmes - BRL 279 compared to BRL 779
- and, in contrast to classroom-based programmes, fees have declined - by 19.8% between
2012 and 2017 (HOPER, 2017;39)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 83

Figure 3.10. Enrolment in distance education programmes (2009-15), by sector

HE *Presential Distance

9000000

8000000

7000000
6000000
5000000
4000000
3000000
2000000
1000000
0 1 1 n n n n n 1

2009 2010 2011 2012 2013 2014 2015 2016 2017

Source: (INEP, 2018,38]), Numero de Matriculas - Educagao Superior - Por Modalidade de Ensino, Grau
Académico e Categoria Administrativa (Enrolment - Higher Education - by sector and category),
http://inep.gov.br/inep-data.

Equity 4and the socio-economic profile of the student population

Greater levels of equity but important gaps remain

Access to higher education has become more equitable in recent years. The share of
students from families in the bottom income quartile who attend higher education increased
from 9.7% in 2003 to 18.8% in 2013 (Nascimento and Verhine, 2017,19}). Conversely, the
share of students from the top quartile has decreased by 17.5 percentage points in the same
period.

Despite improvements, as Figure 3.11 shows, individuals from disadvantaged backgrounds
are still much less likely to participate in higher education. Moreover, as previously
discussed, despite coming from poorer backgrounds, they are less likely to attend public
institutions, which do not charge tuition fees. Less than 60% of students enrolled in public
institutions graduated from a private upper secondary school, compared to70% of those
attending private institutions (INEP, 2018,40)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

84 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

Figure 3.11. Share of enrolment in higher education institutions, by income quartile and

sector
Public sector Private sector
% Hs 2003 ©2013 % T2003 ©2013

70 70
60 60
50 50

Oo «
40 40
30 30
20 20
10 10

: Qt Q2 Q3 Q4 : Q1 Q2 Q3 Q4

Note: Q1 refers to the 25% poorest individuals from the total population, whereas Q4 refers to the 25% richest.
Source: (Nascimento and Verhine, 2017{19]), Consideracgées sobre o investimento publico em educacao superior
no Brasil (Reflections about public investment in higher education in Brazil), Radar: tecnologia, produgao e
comercio exterior, Instituto de Pesquisa Econdémica Aplicada,
http://www. ipea.gov.br/portal/images/stories/PDFs/radar/170324 radar _49.pdf.

Ethnic gaps in access to higher education have declined

Efforts to reduce inequities in access across ethnic groups, through affirmative action, have
led to a significant increase in participation by black and mixed race students. The number
of mixed race (pardo) graduates from federal universities increased from 28.3% to 37.7%
between 2004 and 2014. The share of black graduates also increased in that period from
5.9% to 9.8% (INEP, 2016;36)).

Distance education programmes attract more mature students

There is a large share of mature students (over the age of 30) enrolled in higher education.
This share is more than twice as large in distance programmes (Figure 3.12). Whereas over
50% of students in on-campus programmes are aged 19-24.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 85

Figure 3.12. Share of students in higher education by mode of study and age (2015)

% @Non-distance education Distance education

10
, _. n J a | _ , 1 a 1

18 or less 19-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65+

Source: (SEMESP, 2017,17}), Mapa do Ensino Superior no Brasil 2017 (Map of Higher Education in Brazil
2017), Sindicato das Mantenedoras de Ensino Superior.

Learning and employment outcomes

Tertiary graduates benefit from better employment prospects

In part due to the small share of tertiary-educated people in the population, those who do
obtain a tertiary degree in Brazil can expect a notably higher earnings advantage than on
average across OECD countries. Someone with a bachelor’s degree in Brazil earns over
2.4 times what someone who only attained upper secondary education earns (OECD
average: 1.5) and someone with a master’s, doctorate or equivalent earns almost 4.5 times
more (OECD average, 2.0). These very large earnings premiums are common in Latin
American countries with available data (Chile, Colombia, Costa Rica and Mexico) (OECD,
2017;20)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

86 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

Figure 3.13. Relative earnings of adults, by educational attainment (2016)

25-64 year-olds with income from employment; upper secondary education = 100

Earning advantage* of adults with tertiary education DEarning disadvantage™* of adults with below upper secondary education

Index

SE WW _ OL _B aW NUN Doh oh GY & g ov AAV Dd o& OV. A vos A
FS PPP GFW _ LS. GW GF SG WW GF S) SE OO DN No “BO BAD EE “28 AN
FE LPP PFGF Pf GFF LF ELF FG GFP FW FPG wrewqF FFP COM
we WF wr? ic) Vrs eS yy RIG yr aS ONG g Se oe RS co STEP SEP SF Oe
3s Gd RS Rt RK we Oe
s rw s 9 3 Ss ws
S sO ca
9 A

Note: 1. Year of reference 2015. 2. Index 100 refers to the combined ISCED levels 3 and 4 of the educational
attainment levels in the ISCED 2011 classification. 3. Year of reference 2014. 4. Earnings net of income tax.
5. Year of reference 2012

Source: (OECD, 2018)3)), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2018-en.

As in most OECD and partner countries, those with a tertiary degree in Brazil have better
employment rates overall. At 6.5% in 2016, the unemployment rate for tertiary-educated
adults in Brazil was about over four percentage points lower than for those who attained
only upper secondary education (10.9%), and the inactivity rate was 50% lower (8%
compared to 16% respectively). Again, these differences are much larger in Brazil than on
average in OECD countries (OECD, 2017)20)). However, there is some evidence of a
mismatch between the supply of graduates and the skills required in the labour market.
Nearly 70% of managers in Brazil reported difficulty in filling positions, a larger share than
in Argentina (41%), Costa Rica (40%) and Mexico (38%). According to the survey, the
main reasons reported are lack of technical skills, lack of professional experience and
insufficient number of applicants (McKinsey Global Institute, 2018 (41))

Notes

' The gross enrolment rate is calculated by dividing the total population of enrolled students,
regardless of their age, by the total national population in the age range that typically attends higher
education. Net enrolment is calculated by dividing the number of students in the age range that
typically attends higher education by the total national population of that specific age group.

> New programmes and increases in the number of study places in existing programmes in the fields
of law, medicine, dentistry, psychology and nursing are subject to prior authorisation by MEC for
all institution types. For law programmes, MEC consults the Federal Council of the Brazilian Bar
Association (Ordem dos Advogados do Brasil) and for programmes in medicine, dentistry,
psychology and nursing, the National Health Council (Conselho Nacional de Saude) (Presidéncia
da Republica, 2017, p. art 41,2)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 87

References

CAPES (2018), GEOCAPES - Sistema de Informacées Georreferenciadas (GEOCAPES -
Georeferenced Information System), CAPES, https://geocapes.capes.gov.br/geocapes/
(accessed on 01 August 2018).

CGEE (2016), Mestres e doutores 2015 - Estudos da demografia da base técnico-cientifica
brasileira (Master's and doctoral graduates 2015 - a demographic study of the Brasilian
technical and science base), Centro de Gestao e Estudos Estratégicos, Brasilia,
http://www.cgee.org.br. (accessed on 25 September 2018).

Clarivate Analytics (2018), Research in Brazil A report for CAPES by Clarivate Analytics,
https://www.capes. gov. br/images/stories/download/diversos/17012018-CAPES-
InCitesReport-Final.pdf (accessed on 01 August 2018).

Corbucci, P. and P. Fonseca Marques (2003), Fontes de Financiamento das Instituigdes Federais
de Ensino Superior: um Estudo sobre a Universidade de Brasilia (Funding sources in federal
higher education institutions: a study on the University of Brasilia), IPEA, Brasilia.

Davies, N. (2017), “O Financiamento Public as Instituigdes Privadas de Ensino (Public funding
to private educational institutions)”, Revista Agenda Social, Vol. 10/1.

de Magalhaes Castro, M. (2015), “Higher Education Policies in Brazil: A Case of Failure in
Market Regulation”, in Schwartzman, Simon Pinheiro, Romulo Pillay, P. (ed.), Higher
Education in the BRICS Countries - Investigating the Pact between Higher Education and
Society, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-017-9570-8 14.

Dwyer, T. et al. (2016), Jovens universitarios em um mundo em transformacao : uma pesquisa
sino-brasileira (Young university graduates in a transforming world: a Chinese-Brazilian
study), Ipea, Brasilia,
http://www.ipea.gov.br/agencia/images/stories/PDFs/livros/livros/160715_livro_jovens_univ
ersitarios.pdf (accessed on 03 September 2018).

Ferreyra, M. et al. (2017), At a Crossroads: Higher Education in Latin America and the
Caribbean, World Bank, Washington D.C., http://dx.doi.org/10.1596/978-1-4648-1014-5.

HOPER (2017), Webinar HOPER-Presente e Futuro das Pequenas e Médias IES no Brasil
(Present and future of small and medium-sized HEIs in Brazil),
http://sys.hoper.com.br/webinar/WEBINAR-Mercado%20Educacional-ASBR2017.pdf
(accessed on 17 September 2018).

IBGE (2014), Pesquisa Nacional por Amostra de Domicilios - PNAD (National Household
Survey), Instituto Brasileiro de Geografia e Estatistica, https://www.ibge.gov.br/estatisticas-
novoportal/sociais/populacao/9 127-pesquisa-nacional-por-amostra-de-
domicilios.html?=&t=o-que-e (accessed on 27 August 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

[31]

[33]

[35]

[16]

[18]

[1]

[8]

[15]

[39]

[10]

88 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

IMF (2017), Technical Assistance Report-Supporting Implementation of the Expenditure Rule
Through Public Financial Management Reforms, International Monetary Fund. Fiscal Affairs
Dept..

INEP (2018), Numero de Matriculas - Educagao Superior - Por Modalidade de Ensino, Grau
Académico e Categoria Administrativa (Enrolment - Higher Education - by sector and
category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo (accessed on
12 November 2018).

INEP (2018), Numero de Matriculas da Educacgao Superior Por Categoria Administrativa e
Abrangéncia Geografica (Enrolment in Higher Education, by region and administrative
category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo.

INEP (2018), Relatorio do 2° ciclo de monitoramento das metas do plano nacional de educagao
(Report on the second cycle of monitoring of the goals of the National Education Plan),
INEP, Brasilia.

INEP (2018), Sinopses Estatisticas da Educagdo Superior 2017 — Graduagdao - INEP (Statistical
Sinopsis for Higher Education 2017), http://portal.inep.gov.br/web/guest/sinopses-
estatisticas-da-educacao-superior (accessed on 13 November 2018).

INEP (2016), Sinopses Estatisticas da Educagdo Superior — Graduacgao - INEP (Sinopsis Higher
Education Statistics - Undergraduate Education - INEP),
http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior (accessed on
17 September 2018).

McCowan, T. (2007), “Expansion without equity: An analysis of current policy on access to
higher education in Brazil”, Higher Education, Vol. 53/5, pp. 579-598,
http://dx.doi.org/10.1007/s10734-005-0097-4.

McKinsey Global Institute (2018), Preparing Brazil for the Future of Work: Jobs, Technology,
and Skills - Briefing Note,
https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Future%20o0f%200rgan
izations/Preparing%20Brazil%20for%20the%20future%200f%20work%20Jobs%20technolo
gy%20and%20skills/MGI-Future-of-W ork-Brazil-Briefing-note.ashx (accessed on
17 September 2018).

MEC (2018), Estrutura Organizacional - Ministério da Educagao (Organisational Structure -
Ministry of Education), http://portal.mec.gov.br/institucional/estrutura-organizacional
(accessed on 31 July 2018).

MEC (2018), Qual a diferenga entre pos-graduacao lato sensu e stricto sensu? (What is the
difference between lato sensu and stricto sensu postgraduate programmes ?), Minisério da
Educacao, http://portal.mec.gov.br/component/content/article?id=13072:qual-a-diferenca-
entre-pos-graduacao-lato-sensu-e-stricto-sensu (accessed on 31 July 2018).

[23]

[38]

[37]

[25]

[40]

[36]

[21]

[41]

[14]

[32]

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 3. HIGHER EDUCATION IN BRAZIL | 89

MEC (2017), Portaria N° 468, de 3 de Abril 2017 - dispée sobre a realizagado do Exame
Nacional do Ensino Médio - Enem, e da outras providéncias (Decree N° 468, 3 April 2017,
on the undertaking of the Upper Secondary National Exam - ENEM and other matters),
http://www.lex.com.br/legis_27370339 PORTARIA N_ 468 DE 3 DE ABRIL DE 2017.a
spx (accessed on 10 November 2018).

MEC (2014), Planejando a Proxima Década Conhecendo as 20 Metas do Plano Nacional de
Educagdo (Planning the next decade: get to know the 20 goals of the National Education
Plan), Ministério da Educagao, Brasilia,
http://pne.mec. gov.br/images/pdf/pne_conhecendo_20_metas.pdf (accessed on
08 November 2018).

MEC (2018 (unpublished)), Censo da Educagao Superior 2018 (Higher Education Census
2018), Data supplied by MEC on 5 October 2018.

Nascimento, P. and R. Verhine (2017), “Consideragées sobre 0 investimento publico em
educacao superior no Brasil (Reflections about public investment in higher education in
Brazil)”, Radar: tecnologia, produgao e comeércio exterior, Instituto de Pesquisa Econémica
Aplicada, http://www. ipea.gov.br/portal/images/stories/PDFs/radar/170324 radar_49.pdf
(accessed on 05 September 2018).

OECD (2018), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2018-en.

OECD (2018), OECD Economic Surveys: Brazil 2018, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eco_surveys-bra-2018-en.

OECD (2017), Education at a Glance 2017: OECD Indicators, OECD Publishing, Paris,
http://dx.doi.org/10.1787/eag-2017-en.

OECD (2017), How's Life? 2017: Measuring Well-being, OECD Publishing, Paris,
http://dx.doi.org/10.1787/how_life-2017-en.

OECD (2016), OECD Science, Technology and Innovation Outlook 2016, OECD Publishing,
Paris, http://dx.doi.org/10.1787/sti_in_outlook-2016-en.

OECD (2014), Investing in Youth: Brazil, Investing in Youth, OECD Publishing, Paris,
http://dx.doi.org/10.1787/9789264208988-en.

OECD Data (2018), Jncome inequality, https://data.oecd.org/inequality/income-inequality.htm.

Portal da Camara dos Deputados (2017), “Camara rejeita PEC que autorizava universidade
publica a cobrar por curso lato sensu (The House rejects proposal for constitutional
amendment that would authorise public universities to charge for latu sensu degrees)”,
http://www2.camara.leg.br/camaranoticias/noticias/EDUCACAO-E-CULTURA/528130-
CAMARA-REJEITA-PEC-QUE-AUTORIZAVA-UNIVERSIDADE-PUBLICA-A-
COBRAR-POR-CURSO-LATO-SENSU.html (accessed on 01 August 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

[29]

[4]

[2]

[19]

[3]

[5]

20]

12]

34]

[7]

[9]

[22]

90 | CHAPTER 3. HIGHER EDUCATION IN BRAZIL

Presidéncia da Republica (2014), Lei No. 13005 - Aprova o Plano Nacional de Educagao (PNE) [24]
(Law No. 13005 - Approval of the National Education Plan),
http://www.planalto.gov.br/CCIVIL_03/ Ato2011-2014/2014/Lei/L13005.htm (accessed on
04 September 2018).

Presidéncia da Republica (2007), Decreto n° 6096 - Institui o Programa de Apoio a Planos de [26]
Reestruturagao e Expansao das Universidades Federais - REUNI. (Decree No 6096 creating
the Programme for Support to Restructuring and Expansion Plans for Federal Universities),
http://www.planalto.gov.br/ccivil_03/ Ato2007-2010/2007/Decreto/D6096.htm (accessed on
08 November 2018).

Salto, D. (2018), “To profit or not to profit: the private higher education sector in Brazil”, Higher [6]
Education, Vol. 75/5, pp. 809-825, http://dx.doi.org/10.1007/s10734-017-0171-8.

Schwartzman, J. (2003), O Financiamento das Instituigdes de Ensino Superior no Brasil [30]
(Funding Higher Education Institutions in Brazil), IESALC - UNESCO,
http://www.iesalc.unesco.org. veIES/2003/ED/PI/15 (accessed on 17 September 2018).

SEMESP (2017), Mapa do Ensino Superior no Brasil 2017 (Map of Higher Education in Brazil [17]
2017), Sindicato das Mantenedoras de Ensino Superior.

TCU (2016), Auditoria Operacional no Fundo de Financiamento Estudantil (FIES) (Operational (281
Audit of the Student Financing Fund (FIES)), hittps://portal.tcu.gov.br/biblioteca-
digital/auditoria-operacional-no-fundo-de-financiamento-estudantil-fies.htm.

Transparency International (2018), Corruption Perceptions Index 2017 - Transparency [11]
International,
https://www.transparency.org/news/feature/corruption_ perceptions index_2017 (accessed on
01 August 2018).

World Bank (2006), Brazil Crime, Violence and Economic Development in Brazil - Elements for 131
Effective Public Policy, Report No. 36525-BR,
http://documents.worldbank.org/curated/en/10400146823 1894133/pdf/365250BR.pdf
(accessed on 27 August 2018).

Zalaf Caseiro, L. (2016), Desigualdade de acesso a educagdao superior no Brasil e o Plano [27]
Nacional de Educagao (Inequality in access to higher education in Brazil and the National
Education Plan), NEP, Brasilia.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 91

4. Market entry: New institutions and undergraduate programmes

This chapter focuses on the processes in place in Brazil to regulate the establishment of
new private higher education institutions and of new undergraduate programmes in new
and existing institutions in the federal higher education system. The establishment of new
private institutions and the creation of new undergraduate programmes in all types of
institution require higher education providers to seek regulatory approval from the
Ministry of Education (MEC). This approval depends on compliance with administrative
procedures and a positive outcome from external peer reviews designed to assess the
quality of proposed new institutions and new and recently established programmes. The
chapter analyses these regulatory and evaluation processes, highlighting strengths and
weaknesses, and provides recommendations for their improvement.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

92 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

4.1. Focus of this chapter

This chapter focuses on the processes in place in Brazil to regulate the establishment of
new higher education institutions (HEIs), and of new undergraduate programmes in both
new and existing institutions in the federal higher education system.

Specific processes exist for institutional accreditation for private institutions...

Public HEIs in Brazil are created by federal, state or municipal governments, through
legislation that automatically conveys authorisation to operate for the institution in
question. As explained earlier in this report, once established, public institutions created
and funded by the federal authorities form part of the federal higher education system. They
are then legally subject to regulation and supervision by the Ministry of Education (MEC)
and institutional and programme evaluation by INEP as part of the National System of
Higher Education Evaluation (SINAES). Public institutions created and funded by state or
municipal governments (which include some large institutions such as the University of
Sao Paulo (USP) or the University of Campinas (UNICAMP) in the State of SAo Paulo) are
regulated by state governments and are not obligated to follow the rules of SINAES.

All private higher education providers are considered part of the federal system and are
required to obtain formal external accreditation (credenciamento) from MEC to allow them
to begin operation.

HEIs in Brazil are legally classified as colleges (faculdades), university centres or
universities. In broad terms, colleges are undergraduate teaching institutions; university
centres are required to have more extensive undergraduate provision, more permanent staff
and developed outreach activities; and universities, in addition to meeting these criteria,
must have research activity and provide postgraduate education. Public HEIs created by
federal, state, or municipal governments may be established in any of these three legal
forms from the outset. In contrast, all new private institutions must first be established as
colleges and may subsequently transition to the status of university centre or university if
they meet the relevant criteria and successfully undergo a new external re-accreditation
process. The specific criteria that private HEIs must meet to qualify for each institutional
status (organizacao académica) are set out in articles 16 and 17 of a 2017 decree governing
quality assurance processes in the federal higher education system (Presidéncia da
Republica, 2017;1)).

...and approval of new programmes

As a general rule, colleges in the federal higher education system, which are almost
exclusively private!, are required to obtain formal authorisation from MEC (autoriza¢ao)
to start new programmes. All applications for the establishment of a new higher education
institution must be accompanied by at least one (and up to five) application(s) to create a
new undergraduate programme. Existing colleges must submit applications for
authorisation for each new programme they wish to create. As discussed below, the
complexity of the procedures followed to obtain authorisation varies depending on the
institutional quality score held by the college submitting the request for authorisation and
field of study of the proposed programme. University centres (all of which are private) and
universities” have a greater degree of autonomy and are not generally required to obtain
authorisation in advance to start new programmes, but must notify MEC of the creation of
all new programmes. Universities and university centres do require prior authorisation to
start new programmes in medical fields and law (see below).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 93

All HEIs in the federal system, whatever their legal form, are required to submit new
programmes to an external quality assurance process called “recognition”
(reconhecimento), once half of total teaching hours have been completed (in the second or
third year, for example). All programmes offered by HEIs need to complete the recognition
process successfully for the degrees they award to be valid in Brazil. As a result,
reconhecimento is effectively part of the initial approval process for programmes. Failure
to obtain recognition would mean that the diplomas obtained by graduates would not be
valid nationally and may thus lead to the closure of the programme.

The approval of new institutions and programmes forms a key part of the wider regulatory
work of Secretariat for Regulation and Supervision of Higher Education (SERES) in the
Ministry of Education (MEC), the full scope of which is illustrated in Table 4.1 below.

Table 4.1. Regulatory acts by SERES in 2017

Regulatory Acts Number
Regulatory acts relating to HEIs
Credenciamento (Accreditation) 208
Credenciamento EaD (Accreditation, Distance Provider) 70
Recredenciamento EaD (Re-accreditation, Distance Provider) 36
Transferéncias de Mantenga (Transfer of Ownership) 273
Regulatory acts relating to programmes
Autorizagao nao-vinculada (Authorisation, not linked to accreditation) 1361
Reconhecimento (Programme Recognition) 1686
Renovagao de Reconhecimento (Renewal of Programme Recognition) 6781
Autorizagao EaD (Authorisation of New Programme, Distance Education) 225
Reconhecimento EaD (Renewal of programme recognition, distance education) 266
Renovagao de Reconhecimento EaD (Renewal of Programme Recognition, Distance Education) 188
Aditamento de polos (Addition of Distance Education Sites) 88
Aumento de vagas (Authorisation for increase in vagas) 267
Other regulatory acts
Aditamento (Modification of existing regulatory acts) 645
Chamamento Publico (Public call) 2

Total Acts 12 096

Source: SERES, “Regulagdo e Supervisdo da Educagao Superior” p.17, December 2017, presentation to OECD
Review Team.

4.2. Strengths and weaknesses of the current system

This section analyses the strengths and weaknesses of the current systems of regulation and
evaluation for the accreditation of new private providers and the authorisation and
recognition of new undergraduate programmes.

Establishment of new institutions: accreditation

Relevance: the rationale and objectives of the current system

In all systems of external quality assurance, processes of “market entry” by new higher
education institutions must take care to balance quality and quantity. Entry requirements
must be sufficiently restrictive to prevent bad programmes, such as those in which large
numbers of students are unable to complete their studies, or complete their studies but
acquire few skills and have poor prospects for employment. At the same time, these

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

94 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

requirements must not create barriers to entry that unnecessarily limit supply and leave
students without access to beneficial study opportunities. In addition, badly designed
arrangements for the establishment of new institutions may require new institutions to
adopt governance, management, or staffing arrangements that hamper innovation and
efficiency.

Brazil’s system of accreditation (credenciamento) for new private HEIs exists to ensure
that new institutions meet minimum standards of quality in providing higher education
programmes to undergraduate students. The arrangements for the establishment of new
private HEIs, which must always be coupled with the creation of programmes, strike a
balance between ex ante and near-term ex post review to assure quality. Like other higher
education systems, Brazil applies a lower level of scrutiny to institutions established by
public authorities - in practice, institutions established by federal, state and municipal
governments. Public institutions are not subject to the same accreditation requirements as
private institutions, may be created as universities or university centres, and are exempt
from any other forms of ex ante review. Public law authorises heightened control over the
creation of private provision, permitting newly established private institutions to take only
the form of “college” (faculdade), rather than fully developed university institutions.

Effectiveness: quality indicators used and division of responsibilities

MEC necessarily carries out accreditation of new private institutions based on planned
provision rather than functioning programmes and institutions, and thus must focus on the
planned inputs and processes, rather than observed processes, outputs or outcomes. New
private higher education institutions begin the process of accreditation when a legal
representative of the maintaining institution submits to SERES documents outlining
various dimensions of the planned institution and a self-evaluation of its expected
performance. The information submitted (Presidéncia da Republica, 2017, pp. art. 20-21,1))
includes:

1. Information about the legal, financial and tax status of the operating organisation
(mantenedora).
2. An Institutional Development Plan (PDI), containing:
i. A description of the institutional mission, goals and profile,
ii. An institutional pedagogical programme describing educational, research
and outreach policies,
ili. Details of the number and nature of planned study programmes, including
the number of study places (vagas),
iv. Details of (planned) teaching staff,
v. Planned digital document and records management systems,
vi. Details of physical infrastructure, including library, laboratories etc., where
relevant,
vii. Details of financial capacity and a financial sustainability plan,
viii. Where relevant, specific information relating to distance education.
Internal statutes.
4. Information about the proposed management staff, including their academic
experience.
5. Proof that suitable premises and facilities (im6vel) are available, including technical
appraisal of their accessibility for students with special needs and compliance with
fire and safety regulations.

ies)

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 95

Institutional accreditation is linked to a review of proposed programmes. Private
institutions seeking initial accreditation have up to three years following approval of their
initial institutional plan to submit and obtain authorisation of between one and five
programmes (autorizacdo vinculada a credenciamento). Additionally, campuses that are
located outside of the municipality in which a private institution has its seat of operations
must undergo a separate process of accreditation.

Following review and approval by SERES of documents submitted to it through the ee MEC
online platform, a process of peer review is undertaken. Three external evaluators drawn
from INEP’s BASis database undertake an on-site inspection and review visit, evaluating
the conditions for the proposed institution. This institutional review may be combined with
review of up to two programmes (see below) by the same review commission (Presidéncia
da Republica, 2017, p. art.19j1)).

For the institutional review, the evaluation commission uses specific evaluation criteria and
scoring detailed in an evaluation template (“instrument”) for accreditation developed by
INEP (INEP, 2017;2)). The evaluation process implemented by peer reviewers is organised
around five axes, and assesses the proposed institution against 45 qualitative indicators,
each of which is evaluated on a five-point scale. Some of the indicators apply only to
distance education and others only to physical campuses, meaning a given institution is
only ever assessed against a maximum of 41 indicators.

Of the 45 indicators in total, 30 could best be classified as planned inputs to institutional
management and the educational process (general institutional policies, staffing,
infrastructure and equipment) and another 15 as planned processes (policies for more
specific institutional processes and proposed activities). For obvious reasons, real outputs
cannot be considered in this form of ex-ante evaluation. The principal foci of institutional
accreditation are the Institutional Development Plan (PDI) (30%), planned academic
policies (20%), planned management policies (20%) and infrastructure (20%). The final
score generated by this evaluation, on a scale of one to five, is referred to as the
“institutional score” or Conceito Institucional (Cl). Institutions need a score of at least three
to receive accreditation from SERES.

Table 4.2. Indicators used for on-site inspections for institutional accreditation

Axis Number of indicators Weight
Planning and institutional evaluation 8) 10
Institutional development T* 30
Academic policies 10 20
Management policies 7 20
Infrastructure 18** 20
Total 45 100

Note: * 2 indicators apply only to distance education institutions; ** 4 indicators apply only to distance
education institutions and 4 only to campus-based institutions.

Source: OECD calculations based INEP (2017) External institutional evaluation instrument - classroom-based
and distance - accreditation, adopted by INEP in October 2017 (INEP, 2017,2)).

According to SERES, 829 institutional accreditation reviews were performed in the period
2015-2017, and on average two to three courses associated with each institutional proposal
were reviewed (2 013 courses in total). SERES promulgated 208 acts of accreditation in
2017. It has not reported the number of accreditation proposals that it modified or rejected.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

96 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

Institutional accreditation for private colleges is valid for a period of three to five years;
depending on the CI score (three to five) they receive. After this period, colleges must
undergo a process of re-accreditation (recredenciamento), which formally applies to all
types of private and public institution, and which we discuss in Chapter 7.

Effectiveness and efficiency: use and effects of the current system

The cost of institutional accreditation falls exclusively on private higher education
institutions, since public institutions are exempt from its requirements. Private institutions
bear the direct financial costs of complying with accreditation requirements, including costs
for staff and renting premises in the period between submission of an accreditation request
and approval to begin operations.

Overall, however, the institutional accreditation requirements in place since the adoption
of the 2004 legislation on SINAES do not appear to have created excessive barriers to the
market entry of private higher education providers in Brazil. As highlighted in Chapter 3,
data on enrolment and the number of HEIs show that Brazil’s higher education system has
grown swiftly over the last decade, and private sector institutions have provided the
majority of new study places. Moreover, higher education leaders with whom the review
team met, including representatives of private institutions, did not report that accreditation
requirements resulted in unmet demand among prospective students.

Well-designed accreditation of new higher education providers brings benefits to society,
protecting students, as consumers, from poor or fraudulent provision. In contrast to some
other countries in the Latin America and Caribbean region (IESALC, 2017;3}), compliance
with Brazil’s system of institutional accreditation appears to be nearly universal. Private
institutions do not frequently operate without institutional accreditation. Moreover, the
requirements of institutional accreditation appear to be sufficiently rigorous to limit
fraudulent or grossly unqualified private institutions from entering the higher education
marketplace.

Nonetheless, there are examples of accredited higher education institutions offering
programmes that are not authorised, and organisations that are not accredited higher
education institutions offering fraudulent diplomas (Governo do Brasil, 2017{4}). While the
Ministry’s e-MEC platform provides a single national registry of accredited institutions and
authorised programmes, it is primarily an administrative database. Incidents of allegedly
fraudulent provision suggest that not all students have ready access to information that
allows them to confirm the validity of the institutions and programmes in which they plan
to study. While the layout and functionality of the e-MEC site are not designed to be used
by students and their families, the information contained in the system could easily be
exploited as part of a more user-friendly information service for students.

There are specific concerns in Brazil — as in other countries — regarding the expansion of
distance education, including HEIs that only provide distance programmes. Distance
education now accounts for almost 20% of total enrolment in Brazil, with over 90%
provided by the private sector. Private distance education institutions and the programmes
they provide are subject to the same procedures for institutional accreditation (and
programme-level authorisation and recognition — see below) as providers of traditional
classroom-based higher education. A limited number of qualitative indicators relating
specifically to distance education have been incorporated into the evaluation templates used
for accreditation, covering pedagogical approaches, digital technologies and infrastructure.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 97

Brazilian legislation requires distance education programmes to respect the requirements
of national curriculum guidelines (DCNs), for fields where these exist, and distance
programmes have hitherto mostly been blended programmes, with some face-to-face
instruction and assessments, often conducted in decentralised distance education learning
centres (referred to as “poles”). This pattern might be considered to be positive, as
internationally, blended programmes have been shown to be more effective than fully
online programmes (Escueta et al., 2017,5)).

However, recent legislative changes have made it easier for private higher education
providers to establish large numbers of distance education “poles” (up to 250 a year), in
multiple locations, without the need for the facilities in each location to be inspected by
INEP evaluators (Presidéncia da Republica, 2017(6); MEC, 2017;7)). Some stakeholders in
Brazil are concerned that this will promote the uncontrolled expansion of distance
education, without adequate quality guarantees (Estadao, 2018(7)). Furthermore, the
specific evaluation criteria for distance education institutions (and programmes) used
currently are few in number and underdeveloped in light of the risks associated with this
kind of provision (limited staff-student interaction, the risk students are isolated, the
challenges of organising fair and rigorous assessments and examinations, etc.).

Creation of new programmes: authorisation and recognition

Relevance: the rationale and objectives of the current system

The principal focus of quality assurance with respect to new provision or “market entry” in
Brazil is the offer of new undergraduate study programmes by private HEIs. Recent data
show that the majority of new programmes are created in the private sector. Indeed,
between 2015 and 2016, the number of registered classroom-based and distance
undergraduate programmes in private HEIs increased by 1 092 to 23 824 (a 5% increase),
while the number of programmes in federal universities fell by 161 to 4 876 (a 3% fall)
(INEP, 201 8j6)).

MEC closely supervises the establishment of programmes through processes of ex-ante
authorisation (autorizagdo) for a proportion of new programmes in colleges and
recognition (reconhecimento) of all new programmes in all types of HEI (in practice,
mostly in private HEIs), once they have completed at least half the teaching hours of their
first cycle of operation.

In principle, new courses enter the higher education system after careful and integrated
scrutiny: on-site reviews at the stage of authorisation (in some cases) and recognition (in
all cases) examine the conditions in place that affect the supply of the programme. These
initial checks of the supply conditions for programmes remain the primary guarantor of the
quality of undergraduate programmes in Brazil. They are later complemented by ongoing
monitoring of a small set of indicators, based primarily on the results of assessment of
student learning outcomes using the National Examination of Student Performance
(ENADE) and periodic “renewal of programme recognition” that follows a three-year
cycle. In cases where quality problems are detected through the monitoring indicators, or
through other sources, such as complaints, INEP conducts further on-site visits for renewal
of recognition. We discuss these later quality checks in Chapter 5.

Authorisation

Colleges must seek authorisation of all new programmes before instruction begins. The
standard authorisation process starts with analysis by SERES of documents relating to the

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

98 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

new programme submitted by the HEI. These documents include a “Programme
Pedagogical Project” (projeto pedagdgico do curso, PPC), setting out the programme
structure, number of study places (vagas), proposed curriculum, teaching methods and use
of technology. In addition, HEIs provide details of proposed teaching staff and proof of
available teaching spaces, including, where relevant, decentralised learning centres
(“poles”) for distance education (Presidéncia da Republica, 2017, p. art.43;1)).
Subsequently, in the standard authorisation procedure, an on-site review is organised by
INEP. This review is undertaken by external reviewers from the BASis database, using the
dedicated evaluation template for authorisation (INEP, 2017(7)), discussed below. The
evaluation report and a score, referred to as a Conceito de Curso (CC) and awarded on a
scale of one to five, are then transmitted to SERES. Finally, SERES reviews the evaluation
report, approves the CC and makes a decision to authorise the programme, if the CC is
three or above, or to refuse authorisation.

The authorisation of new programmes proposed by colleges is a risk-adjusted process.
Recent changes to the regulatory regime allow colleges to obtain authorisation for new
courses under certain circumstances based exclusively on a desk-based analysis by SERES
of the programme documents submitted by the HEI, without undergoing an on-site
inspection. Colleges with the minimum institutional quality score (CI) of three can start up
to three new programmes a year without on-site reviews, provided they already have
officially recognised (i.e. quality assured) programmes in the same disciplinary field.
Colleges with institutional quality scores of four and five are permitted to create,
respectively, up to five and ten new programmes a year under the same lighter regulatory
conditions, in fields where they have existing quality assured programmes (MEC, 2017,s)).

Programmes in law, medicine, dentistry, psychology and nursing form a major exception
to the general principles guiding authorisation. Programmes in these fields now always
require prior authorisation from MEC and an on-site authorisation review, even in
university centres and universities (Presidéncia da Republica, 2017, p. art.41,1)). To inform
its decisions in these fields, MEC takes advice from, respectively, the federal council of the
Brazilian Bar Association (Ordem dos Advogados do Brasil, OAB) and the National Health
Council (Conselho Nacional de Saude). Compared to previous legislation, the 2017 decree
on quality assurance (Presidéncia da Republica, 201711;) extended the requirement for
systematic authorisation with on-site inspections to nursing programmes and also made
increases in the number of study places in all undergraduate law and medicine programmes
dependent on MEC authorisation (SEMESP, 2017}9)).

Recognition

All HEIs, including university centres and universities, must seek regulatory “recognition”
(reconhecimento) from SERES for every undergraduate programme when a programme’s
first cohort of students has completed between 50 and 75 percent of the workload of the
course. Recognition is needed for the diplomas issued to graduates of the programme to be
valid nationally in Brazil. A separate recognition process is required for each programme
offered in campuses outside the municipality where the HEI has its headquarters. Formally,
the process of recognition also applies to federal public institutions. In practice, the low
levels of programme creation in the federal public sector mean federal public institutions
are comparatively rarely involved in processes of recognition.

HEIs are required to submit requests for the recognition of their programmes to SERES,
providing the same set as documents as is required for authorisation of programmes
(programme pedagogical project etc.). SERES undertakes a desk-based analysis of these

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 99

documents and INEP organises an on-site review process by external evaluators, who use
a separate evaluation instrument to rate programmes and generate a programme quality
score (Conceito de Curso, CC) on a scale of one to five (INEP, 2017;10}). For programmes
in law, SERES seeks an opinion from the Brazilian Bar Association, and for programmes
in Medicine, dentistry, psychology and nursing, from the National Health Council, on the
decision to recognise the programme.

Ifthe CC score 1s at least three and, where relevant, the Bar Association and Health Council
issue positive opinions, SERES confirms the CC and issues an official recognition of the
programme. For programmes previously subject to authorisation, the CC resulting from the
recognition process becomes the new quality score for the programme.

If the result of the on-site evaluation is negative (a score of two or less), SERES requires
the HEI to draw up a “Commitment Protocol” (protocolo de compromisso) which sets out
how the quality problems detected will be addressed within a 12-month timeframe
(Presidéncia da Republica, 2017, p. art.541)). This stage of the supervisory process is
referred to as a “remediation procedure” (procedimento saneador). If it considers there is
an immediate risk for students, SERES may also impose one or more sanctions (medidas
cautelares), including suspension of new student intakes (see Box 4.1). Internal data
transmitted to the OECD team by SERES suggests these kinds of sanction are virtually
never applied in remediation procedures.

At the end of the period established by the Commitment Protocol, the programme is subject
to another on-site inspection by INEP evaluators. If it still fails to meet minimum quality
requirements, SERES launches a “sanctioning procedure” (procedimento sancionador),
which may entail the same sanctions mentioned in Box 4.1 (Presidéncia da Republica,
2017, p. art.73,1)). For serious cases in private institutions, the relevant legislation allows
for the withdrawal of institutional accreditation, which would effectively lead to the closure
of the institution. Again, in practice, such cases are rare. Legally, some of the sanctions can
be applied to public institutions, but the legal status of these institutions as public bodies
means they may not have their institutional accreditation withdrawn.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

100 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

Box 4.1. Sanctions used by SERES in supervision of (private) higher education

I - suspension of admission of new students;
II - suspension of the offer of undergraduate or /ato sensu postgraduate programmes;
III - suspension of the autonomy of the HEI;
IV - suspension of the prerogative to create new distance education poles by the HEI;

V - suspension of regulatory processes that the HEI or other HEIs owned by the same
operating organisation have submitted to SERES;

VI — prohibition of filing new regulatory processes to SERES by the HEI or other HEIs
owned by the same operating organisation;

VII - suspension of the HEI’s right to enter into new Student Financing agreements as part
of the FIES system;

VIII - suspension of the HEI’s right to participate in a selective process for the offer of
scholarships from the University for All Programme (PROUND;

IX - suspension or restriction of the HEI’s right to participate in other federal access
programmes.

Source: Article 63 of Decree 9 235 of 15 December 2017 (Presidéncia da Republica, 2017[1)).

Effectiveness: indicators used for authorisation and recognition of new courses

The on-site evaluation templates (“instruments”) used by external reviewers for the
processes of authorisation and recognition were revised by INEP in October 2017. They
establish nearly identical review templates. The judgement criteria in the template for
authorisation (INEP, 2017;7;) focus on planned inputs and activities, while those in the
template for recognition (INEP, 2017j10)) refer to real inputs and activities, verified in
practice by the external review commission sent by INEP.

Both templates direct reviewers to evaluate programmes on three dimensions: the proposed
pedagogical approach and organisation of the programme (Organizacao Didatico-
Pedagogica); the instructional workforce (Corpo Docente e Tutorial); and infrastructure
(Infraestrutura). The assessment of the pedagogical approach and organisation of the
programme considers the extent to which the planned curriculum meets the requirements
of subject-specific National Curriculum Guidelines (Diretrizes Curriculares Nacionais,
DCN) approved by the National Education Council, which exist for many, but not all,
disciplines in higher education. Together, these dimensions contain more than 50
indicators, some of which apply to all programmes, others specifically to distance education
or classroom-based programmes, and still others to programmes offering clinical or field-
based learning. Crucially, peer reviewers are responsible for scoring the indicators on a
five-point qualitative scale.

The 50+ indicators used in each on-site evaluation template focus either on programme
inputs (teaching staff, infrastructure) and processes (pedagogical processes, support to
students etc.). Although both instruments assess the expected “profile” specified for the
graduates that the programmes educate, they do not consider programme outputs. This
choice is necessary because the reviews take place prior to, or midway through, the study
programme for the first student cohort. Other than the distinction between planned and real

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 101

inputs and activities, the principal difference between the two evaluation instruments rests
with the weight they assign to the indicators, with the instructional workforce taking on
greater weight in the process of recognition (40%) than in authorisation (20%).

The indicators assessed under the “teaching staff’ dimension of the evaluation instruments
focus primarily on the qualifications of staff, their employment status and the extent to
which their profiles match the needs of the programme. The choice to assign greater weight
to assessment of these factors during the recognition process than in authorisation reflects
the fact that the staff will actually be in place and working at the time of recognition, so the
composition of the teaching workforce can be judged more accurately. However, the
judgement criteria reward the presence of full-time staff with doctoral degrees and attach
little value to professional experience, thus disadvantaging professionally oriented
programmes. At the same time, relatively little weight is attached to assessment of the
pedagogical and didactic approaches implemented by the programme, despite their crucial
role in supporting students to acquire relevant learning outcomes.

The weighted sum of scores provided in the on-site review are used to calculate the
programme score, the Conceito de Curso (CC), the value of which ranges from one to five,
and provides the basis for SERES to authorise or recognise new programmes, or not.

Table 4.3. Indicators used for authorisation (autoriza¢gdao)

Dimension Number of indicators Weight
Pedagogical and didactic organisation of the programme 24 40
Instructional workforce 14 20
Infrastructure 16 40
54

Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance —
authorisation (INEP, 2017,7}).

Table 4.4. Indicators used for recognition (reconhecimento)

Dimension Number of indicators Weight
Pedagogical and didactic organisation of the programme 24 30
Instructional workforce 16 40
Infrastructure 18 30
58

Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance —
recognition and renewal of recognition (INEP, 2017110).

The on-site evaluation templates now make special provision for the authorisation and
recognition of distance education courses. For example, in evaluating the pedagogical and
didactic organisation of the programme, three indicators consider respectively the tutoring
approach, tutoring staff and virtual learning environment for distance education
programmes (INEP, 2017(7); INEP, 2017,10)). Under the section on instructional workforce,
peer reviewers are also to assess whether programmes have staff with “experience in
teaching in distance education” and “experience of tutoring in distance education” (in this
context, “tutor” is used to refer to teaching assistants, who support core academic teaching
staff in delivering a programme). Infrastructure indicators have, likewise, been augmented
to take account of distance education programmes.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

102 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

However, 45 out of 55 indicators in the templates apply to both classroom-based and
distance programmes. The specific indicators of programme quality related to curriculum,
instruction, learning support, and assessment in distance programmes are less developed
than those used in accreditation systems in other OECD and partner countries, including
the United States (DEAC, 2018 ;11;). Developing appropriate measures of quality that reflect
the specific characteristics of distance education is, however, a challenge shared by many
higher education systems.

Effectiveness and efficiency: use, effects and efficiency of programme
authorisation and recognition

What are the principal effects of programme authorisation and recognition for Brazilian
higher education? First, authorisation and recognition play a critical role in regulating the
enrolment capacity of the nation’s higher education system. Through these processes,
private higher education institutions propose the number of study places they believe
programmes can properly accommodate in their Programme Pedagogical Project (PPC).
The on-site reviews evaluate supporting evidence for this claim?, and their assessment is
confirmed by SERES, which officially determines how many study places the programme
may have.

The impact of the authorisation and recognition processes on the quality of programmes
provided is much more difficult to ascertain. There are no programmes in private
institutions in Brazil that have not been subject to recognition that could form a comparison
group with which to compare recognised programmes in an effort to analyse the quality
effects of the regulatory and evaluation processes. There are no other readily available and
comparable indicators of programme quality in Brazil that would provide an alternative
means to assess programme quality - and make it possible to judge whether recognition
provides an effective guarantee of quality.

The formal requirement for all courses to obtain official recognition in the early stages of
their operation provides a basic guarantee of the quality of programmes. The procedures in
place force higher education providers to reflect seriously about the design of the
programmes they are providing and put in place a range of policies and processes —
described in the programme pedagogical project — that should contribute positively to the
delivery of relevant and high-quality programmes. Nevertheless, the factors verified
through the on-site evaluation at the stage of recognition are all conditions for the delivery
of quality programmes, but do not provide a guarantee that programmes deliver high-
quality education in practice.

Moreover, the processes used to evaluate the quality of new programmes are subject to four
principal lines of criticism with respect to their reliability, usefulness, and cost
effectiveness.

First, representatives of private institutions consulted by the OECD review team
complained that external reviewers appointed by INEP to implement on-site reviews very
frequently come from public universities, while the vast majority of authorisation and
recognition processes occur in private institutions. More generally, institutional
representatives argued that those who are called upon to carry out reviews sometimes lack
expertise with respect to the programme under review. In the first case, reviewers may
bring to private institutions unfamiliarity with their circumstances, or even an active
hostility to their institution. In the second, the risk is that reviewers lack sufficient expert
knowledge to make sound judgements about the didactic and pedagogical profiles of the
programmes they review.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 103

INEP claims that recent improvements to the BASis database of reviewers and the rules
governing the allocation of external experts should address both concerns. The ordinance
governing on-site evaluation requires that review commissions for different types of
programme have direct experience with programmes in the same field and mode of
provision (MEC, 2017;12}). Moreover, a recent “administrative instruction” (MEC, 2017,13))
requires that at least one reviewer in institutional reviews in private HEIs has experience
in a private HEI. It has not been possible to determine, however, to what extent these rules
are applied in practice and, in the latter case, whether experience in the private sector is
required for programme-level reviews in private institutions.

Second, there is concern about the subjectivity or unreliability of qualitative assessments.
The process of on-site review for programme authorisation and recognition (as revised in
late 2017) asks reviewers to make qualitative judgements on a five-point Likert scale, using
pre-formulated judgement criteria. Despite the attempts by INEP to formulate the
judgement criteria clearly, these scales still leave considerable room for interpretation.
They call upon reviewers to make distinctions that that are likely to be inconsistent between
individuals. The OECD review team was told by campus officials that the same programme
offered in different campuses with otherwise near-identical supply conditions received
different marks from on-site reviewers.

Third, the OECD review team heard frequent criticisms from institutional representatives
of the delay and burden associated with the on-site review process for authorisation and
recognition. Describing the whole system of on-site reviews as it functioned in 2012 (before
recent reforms), a document written by the Association of Private Higher Education,
ABMES, notes:

The evaluation system is nearing collapse. INEP holds approximately 5 000
assessment visits per year, or about 100 per week. The logistics to support an
operation of this size, nationwide, and every day is overwhelming. For example,
there are more than 400 flights per week to be scheduled, budgeted, accounted for
and issued by INEP. Yet, for a system with nearly 30 000 undergraduate programs
and 3000 institutions, not counting new authorization and accreditation
procedures for courses and institutions, 5 000 visits are insufficient. This causes
crowding of the evaluation system and a growing backlog. There are higher
education institutions with applications for recognition awaiting for years the visits
of committees. (de Magalthaes Castro, 201514)

INEP and SERES argue that the situation has improved since 2012. In particular, they point
to the fact that HEIs that have received adequate quality scores (a CI of three or above) are
exempted from on-site reviews at the stage of authorisation for programmes in fields where
they already have courses (within certain limits). They argue that the most recent regulatory
changes in Decree 9 235/2017 (Presidéncia da Republica, 2017;1;) reduce burden for
institutions with an established quality record, allowing them to create additional study
places more easily, for example.

While there has indeed been a shift in the regulatory approach, the market entry process for
new undergraduate programmes in the federal higher education system remains
administratively burdensome for private HEIs and the evaluation agency (INEP) when
compared to equivalent processes in many OECD countries. In Brazil, despite the recent
changes, all new programmes are required to go through the recognition process, with on-
site reviews that depend on peer review and are logistically complex to organise.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

104 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

In Anglophone OECD countries and a number of non-Anglophone European higher
education systems, HEIs can create programmes and issue valid diplomas without prior
authorisation (European University Association, 201815])*. In these and other systems,
authorities often link quality review procedures more closely to risk of poor quality than is
the case in the Brazilian system, with less complex procedures in place for institutions that
can demonstrate they present a lower risk. Although the large private higher education
sector in Brazil creates specific risks, which are not found in all higher education systems,
there is certainly scope for Brazil to draw on risk management practice in other quality
assurance systems?.

Finally, on-site visits carried out in support of programme recognition permit higher
education institutions initially to award degrees without providing evidence about the initial
performance of the programme, such as rates of attrition among its students. Additionally,
the process of recognition does not systematically elicit information from the students who
the programmes serve (as the ENADE process does at a later stage), or external
stakeholders who have experience of working with the programme and its students, such
as public sector employers and private firms which provide internships.

4.3. Key recommendations

1. Improve the reliability and visibility of information about institutions’
accreditation status to ensure students and families are well informed

Although MEC, with the support of evaluations coordinated by INEP, regulates the entry
of new institutions into the Brazilian higher education marketplace more comprehensively
than in other systems undergoing rapid expansion, the quality assurance system is not fully
effective in preventing fraudulent and unauthorised provision. The first line of defence
against unaccredited higher education providers is students themselves. Informed students
understand which institutions are accredited and not, and why this matters to them, and are
able to identify and avoid unaccredited institutions. In principle, comprehensive
information about accredited institutions and recognised programmes is available through
the online e-MEC. However, e-MEC is not a user-friendly source of accreditation
information. More accessible public Internet resources found in other higher education
systems could serve as references for the Brazilian authorities in this regard (UK
Government, 201 8;16)). In the medium-term, the aim should be to develop a comprehensive
online portal providing students and prospective students not only with programme-level
information on quality assurance results, but also on issues such as graduation rates and
graduate employment outcomes (see discussion on programme indicators below).

2. Over time, increase the focus on institutions as units of evaluation in the
external quality assurance system to reduce burden, while maintaining
effectiveness

We have noted four main concerns about the processes used to authorise and recognise new
study programmes. Despite attempts to address concerns about the composition of review
commissions and reduce requirements for authorisation in some cases, the Brazilian system
of programme review at market entry remains complex and burdensome and may not
represent the best use of the country’s resources. There is a need for a system in which the
burden and benefit of new programme approval are re-balanced.

Permitting institutions themselves to play a wider role in assuring the quality of
programmes, while maintaining an enhanced system of programme-level monitoring

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 105

indicators, could significantly reduce the burden of programme approval through
authorisation and recognition. Programme-focused regulatory decisions — for new and
existing programmes - account for more than 10 000 of the 12 000 acts that SERES handles
annually.

The Brazilian system of quality assurance currently focuses proportionally more efforts on
the programme-level than on the institutional level as a unit of evaluation and monitoring.
To some extent, the current system regards HEIs as “holding units” for programmes, which
are then the main focus of detailed analysis in the quality assurance system. In Chapter 7
we take up the question of institutional quality, and make suggestions for a more rigorous
and comprehensive process of institutional re-accreditation, with a view to permitting
higher education institutions with demonstrated capacity to assume responsibility for
quality of the programmes that they offer and to become “self-accrediting institutions”.

3. In the near term, take steps to improve the evaluation process for
programmes that remain subject to programme-level authorisation and
recognition

The OECD review team sees a clear case for maintaining programme-level authorisation
and strict market entry requirements at programme-level for HEIs that lack a strong track
record of good quality provision and are not able to demonstrate adequate capacity to self-
accredit their own programmes. It is thus important to increase the effectiveness of these
processes in promoting quality practices for institutions that remain subject to programme-
level authorisation and/or recognition. Priorities for improving current practice in the short-
term include:

e Further improving the criteria used to select and assign peer reviewers for on-site
reviews to increase the fit between reviewer expertise and programme review
responsibilities. It is particularly important that reviewers for professionally
oriented programmes have adequate understanding of the objectives and operation
of such programmes and are able to make robust assessments about the quality of
provision in teaching institutions that lack a traditional academic focus and research
activities.

e Continuing and increasing efforts to improve the training of peer reviewers, with a
view to improving the reliability and impartiality of scoring.

e Increasing the weight attached to the organisation and implementation of teaching
and learning in the evaluation instrument for recognition, reflecting the importance
of these factors for students.

e In cooperation with international peers, refining and expanding the specific
indicators used for the evaluation of distance education programmes, so that these
address the particular risks associated with this type of provision. This should
consider how best to evaluate decentralised distance education centres (“poles”).

e Using the recently introduced process of feedback about the performance of peer
reviewers to monitor and revise selection and training.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

106 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

4. In the longer term, take steps to reduce further the burden and improve the
effectiveness of quality assurance processes for programmes outside self-
accrediting institutions

In the longer term, two issues should be considered in particular. First, the procedures for
on-site visits could be fundamentally reformed. Responsibility for reviewing institutional
infrastructure and basic institutional policies could be assigned to a well-trained and
professionalised inspectorate. The expert judgement of academic peers (who currently
review all aspects of institutions and programmes) could then be applied to a more limited
set of indicators than at present, focused on core teaching and learning activities. A
sequenced process of accreditation and authorisation could be implemented in which a
professional inspectorate initially carried out its work, and academic peers would be
engaged only for institutions and programmes that have passed a first stage of review.
Second, it will be important to identify ways in which the more extensive, quantitative, and
comparable information about intermediate programme performance can be incorporated
into the process of programme recognition. Examples include student attrition from
programmes, and student feedback concerning the teaching and learning environment.

Notes

' Tn 2016, there were 1 866 private colleges (“faculdades’) in Brazil and only four federal public
colleges. In addition, there were 134 state and municipal public colleges, but these are not subject
to the federal regulatory and quality assurance system for institutions and undergraduate
programmes (Source: INEP).

> In 2016, there were 63 federal public universities, 89 private universities and 156 private university
centres in Brazil. No federal public institutions had the status of “university centre”.

3 Peer reviewers use indicator 1.20, “Numero de vagas” (number of study places), to evaluate
programme capacity. An on-site evaluation with a score of three or higher authorises the programme
to have the number of study places proposed in the PPC. The programme may subsequently seek
additional study places by submitted a request to SERES, a new on-site evaluation and another
regulatory act called ““4umento de vagas” (increase in study places).

4 This is the case in Austria, Ireland, Luxembourg, Norway, Poland, Sweden, Switzerland, United
Kingdom (European University Association, 2018>15))

> Australia, for example, has a highly developed approach to risk in quality assurance, outlined by
the Tertiary Education Quality and Standards Agency in their Risk Assessment Framework
(TEQSA, 2018 20))

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 107

References

de Magalhaes Castro, M. (2015), “Higher Education Policies in Brazil: A Case of Failure in [17]
Market Regulation”, in Schwartzman, Simon Pinheiro, Romulo Pillay, P. (ed.), Higher
Education in the BRICS Countries - Investigating the Pact between Higher Education and
Society, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-017-9570-8 14.

DEAC (2018), DEAC Accreditation Handbook, Distance Education Accrediting Commission [14]
(DEAC), Washington, D.C., http://www.deac.org (accessed on 15 November 2018).

Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review”, NBER Working [5]
Paper Series, No. 23744, National Bureau of Economic Research, Cambridge,
http://www.nber.org/papers/w23744 (accessed on 23 November 2018).

Estadao (2018), Governo vai fiscalizar oferta de curso a distancia (Government will investigate [8]
supply of distance education courses), Estadao,
https://educacao.estadao.com.br/noticias/geral,governo-vai-fiscalizar-oferta-de-curso-a-
distancia, 70002327119 (accessed on 02 December 2018).

European University Association (2018), University Autonomy in Europe, [18]
https://www.university-autonomy.eu/ (accessed on 15 October 2018).

Governo do Brasil (2017), MEC identifica irregularidades em cursos de 27 instituigdes de [4]
ensino superior (MEC identifies irregularities in programmes in 27 HEIs), News site of the
Brazilian Government, http://www.brasil.gov.br/noticias/educacao-e-ciencia/2017/06/mec-
identifica-irregularidades-em-cursos-de-27-instituicoes-de-ensino-superior (accessed on
14 November 2018).

IESALC (2017), Educacion superior y sociedad : Aseguramiento de la Calidad en America B]
Latina, Instituto Internacional de Unesco para la Educacion Superior en América Latina y el
Caribe (IESALC), Caracas, http://www.revencyt.ula.ve (accessed on 14 November 2018).

INEP (2018), Sinopses Estatisticas da Educagao Superior 2017 — Graduagao - INEP (Statistical [9]
Sinopsis for Higher Education 2017), http://portal.inep.gov.br/web/guest/sinopses-
estatisticas-da-educacao-superior (accessed on 13 November 2018).

INEP (2017), Instrumento de Avaliagdo de cursos de graduagdo Presencial e a distancia - [13]
Reconhecimento e Renovagdao de Reconhecimento (Evaluation instrument for undergraduate
programmes - classroom-based and distance - recognition and renewal of recognition),

Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, Brasilia,
http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

INEP (2017), /nstrumento de Avaliagdo de cursos de gradugdo - Presencial e a distancia - [10]
Autorizagao (Evaluation instrument for undergraduate programmes - classroom-based and
distance - authorisation), http://www.publicacoes.inep.gov.br (accessed on
11 November 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

108 | CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES

INEP (2017), Instrumento de Avaliagdao Institucional Externa - Presencial e a distancia -
Credenciamento (External institutional evaluation instrument - classroom-based and distance
- accreditation), Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira ,
Brasilia,
http://download.inep.gov.br/educacao_superior/avaliacao_institucional/instrumentos/2017/IE
S_credenciamento.pdf (accessed on 11 November 2018).

MEC (2017), Instrugdo normativa No 2, de 18 de dezembro de 2017 (Normative instruction 2 of
18 December 2017), Ministério da Educagao / Instituto Nacional de Estudos e Pesquisas
Educacionais Anisio Teixeira, Brasilia,
http://portal.imprensanacional.gov.br/web/guest/consulta?p_p_id=l01&p_p_ lifecycle=0&p_p

state=maximized&p_p mode=view& 101 struts _act...1/5 (accessed on
15 November 2018).

MEC (2017), Portaria N° 20, de 21 de dezembro de 2017. (Ordinance 20 of 21 December 2017,
relating to procedures and decision-making standards for the processes of accreditation, re-
accreditation, authorisation, recognition and renewal of recognition of classroom-based and
distance higher education programmes, as well as modifications to these, in HEIs in the
federal education system), http://www.in.g (accessed on 14 November 2018).

MEC (2017), Portaria Normativa No 19, de 13 dezembro de 2017 Dispée sobre os
procedimentos de competéncia do Instituto Nacional de Estudos e Pesquisas Educacionais
Anisio Teixeira - INEP referentes a avaliacao de instituigdes de educagao superior, de cursos
de graduacdao e de desempenho académico de estudantes., Ministério da Educagao, Brasilia,
http://www.abmes.org.br (accessed on 15 November 2018).

MEC (2017), Portaria Normativa no.11, de 20 de junho de 2017 - Estabelece normas para o
credenciamento de instituigdes e a oferta de cursos superiores a distancia, em conformidade
com o Decreto n° 9.057, de 25 de maio de 2017., http://www.in.gov.br/autenticidade.html
(accessed on 04 December 2018).

Presidéncia da Republica (2017), Decreto N° 9.057, de 25 de maio de 2017 Regulamenta o art.
80 da Lei n° 9.394, de 20 de dezembro de 1996, que estabelece as diretrizes e bases da
educagao nacional. (Decree 9057 concerning distance education),
http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9057.htm (accessed on
02 December 2018).

Presidéncia da Republica (2017), Decreto N° 9.235, de 15 de dezembro de 2017 - Dispode sobre o
exercicio das funcgdes de regulagdao, supervisdo e avaliagao das instituigdes de educacgao
superior e dos cursos superiores de graduagdo e de pos-graduagcao no sistema federal de
ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of
regulation, supervision and evaluation of higher education institutions and undergraduate
and postgraduate courses in the federal education system),
http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on
10 November 2018).

SEMESP (2017), Decreto no 9235/17: Andlise comparativa, SEMESP, Sao Paulo,
http://www.semesp.org.br/wp-content/uploads/2017/12/Comparativo-Decretos-21.12.pdf
(accessed on 15 November 2018).

[2]

[16]

[11]

[15]

[7]

[6]

[1]

[12]

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 4. MARKET ENTRY: NEW INSTITUTIONS AND UNDERGRADUATE PROGRAMMES | 109

TEQSA (2018), Risk Assessment Framework - Tertiary Education Quality and Standards [20]
Agency, Tertiary Education Quality and Standards Agency, https://www.teqsa.gov.au/risk-
assessment-framework (accessed on 15 November 2018).

UK Government (2018), Check if a university or college is officially recognised - GOV.UK, [19]
https://www.gov.uk/check-a-university-is-officially-recognised (accessed on
15 November 2018).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 111

5. Assuring and promoting quality for existing undergraduate programmes

This chapter examines the processes in place in the federal higher education system in
Brazil to monitor the quality of established undergraduate programmes and take action in
the event of poor performance. As currently designed, the cycle of ongoing quality
monitoring involves the use of large-scale student testing, administered through the
National Examination of Student Performance (ENADE). ENADE results feed, with other
indicators, into a composite indicator of programme performance — the Preliminary
Course Score (CPC). When programmes score poorly on the CPC measure, they are
subjected to an on-site peer-review visit and may ultimately face sanctions imposed by the
Ministry of Education. The chapter provides a critical assessment of these processes and
provides recommendations for their improvement.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

112 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

5.1. Focus of this chapter

Undergraduate programmes are subject to an ongoing cycle of evaluation and
regulation

Once undergraduate programmes have been recognised under the procedures discussed in
the previous chapter, they are subject to an ongoing cycle of evaluation, coordinated by
INEP on behalf of the Ministry of Education. As currently designed, this cycle involves the
collection and collation, by INEP, of programme-level data, including the results of the
programme’s students in a national assessment of learning outcomes, to create a composite
indicator of programme performance every three years. As a general rule, in cases where
programmes score low ratings in relation to this composite indicator, a new on-site
programme review is undertaken by external evaluators. The results of a programme in
relation to the composite indicator and, where used, the on-site review determine whether
or not SERES renews its official recognition, thus guaranteeing that the diplomas awarded
by the programme retain national validity.

Large-scale, external student testing forms part of this cycle

One of the most distinctive features of this ongoing quality assurance process for
undergraduate programmes in Brazil is the use of large-scale, discipline-specific student
testing. The assessment of the performance of students from every undergraduate
programme is an explicit requirement of the 2004 legislation that established SINAES, the
National System of Higher Education Evaluation (Presidéncia da Republica, 2004,
p. art.2;1}). Each year, students graduating from programmes registered in a particular set
of disciplinary fields are required to take a mandatory competency assessment — the
National Examination of Student Performance (Exame Nacional de Desempenho de
Estudantes, ENADE). Disciplines are assigned to three broad groups, with disciplines in
group I evaluated one year, group II the year after and group II] the year after that, meaning
each discipline is subject to ENADE every three years'. The ENADE tests contain a general
competency assessment common to exams in all fields in a single year and a discipline-
specific component. In addition, all students participating in ENADE are required to
complete a student feedback questionnaire providing biographical information and a
personal assessment of their programme.

The results of ENADE tests feed into a composite indicator of programme
quality

The results achieved in ENADE by students in a given programme are converted into an
average, which is then assigned a score on a scale of one to five, based on its position in
the distribution of the average scores for all programmes in the federal higher education
system in the same disciplinary field. The resulting score out of five is referred to as the
“ENADE score” or Conceito ENADE. The Conceito ENADE is subsequently used,
alongside a standardised indicator based on a comparison of ENADE scores for each
student with their previous performance in the Exame Nacional do Ensino Médio (ENEM)
school-leaving exam; data on staff involved in the programme; and scores from the student
feedback questionnaire, to generate an overall score for each programme on a scale of one
to five. This programme score — misleadingly named the Preliminary Course Score or
Conceito Preliminar de Curso (CPC) - is used to provide an indication — albeit partial - of
the overall performance of the programme in question (MEC, 2017,2)).

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 113

If the CPC is three or above, the programme usually has its official recognition renewed by
SERES (a regulatory process called Renovagdo de reconhecimento), without further direct
evaluation until the subsequent round of ENADE, three years later (MEC, 2017)2)). If a
programme obtains a CPC score of one or two, INEP organises another on-site evaluation
visit, using the same criteria and evaluation template as used for the initial process of
programme recognition, discussed in the previous chapter (INEP, 2017;3)). If the site visit
leads to a positive assessment, with a Conceito de Curso (CC) of three or above, the
programme in question has its recognition renewed. If it obtains a score lower than three,
it will be subject to supervisory measures by SERES and ultimately risks losing formal
recognition — and thus national validity for the diplomas it awards. SERES may request
that programmes that achieve a CPC of three or above are also subject to site visits if there
are other reasons to do this.

The steps in this cycle of evaluation and regulation are illustrated in Figure 5.1 below.

Figure 5.1. The cycle of evaluation and regulation for undergraduate programmes

Renewal of
Recognition First ENADE cycle RecOentlon
(Reconhecimento) (Renovagao de
reconhecimento)

On-site visit leading to new
Conceito de Curso (CC) > 3 =
Renewal of recognition
Conceito ENADE (CE) &
Conceito Preliminar de
Curso (CPC) =< 3

On-site visit leading to new
Conceito de Curso (CC) <3 =

Non-renewal of recognition
Conceito de Curso (CC) 8

Conceito ENADE (CE) & Existing Conceito de Curso
Conceito Preliminar de (CC) retained alongside CPC
Curso (CPC) =>3 = Renewal of recognition

Note: The ENADE cycle for each subject area is repeated every three years.
Source: OECD based on information from INEP and SERES (MEC, 2017,2)).

5.2. Strengths and weaknesses of the current system

The sections which follow provide an assessment of the strengths and weaknesses of the
current model of ongoing quality assurance for undergraduate programmes in Brazil,
involving the systematic use of ENADE and the Preliminary Course Score (CPC) and
selective use of site visits.

Measuring student performance: ENADE

Relevance: rationale and objectives of the current system

Brazil is one of the few countries worldwide that implements a national system of testing
for undergraduate students and is probably the only higher education system that draws on
external assessment of learning outcomes to such a large extent in its regulatory,

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

114 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

supervisory and quality assurance model. The most notable other examples of system-wide
external student testing in higher education in the world are also in Latin America.

Colombia has compulsory competency testing for graduating undergraduate students in the
form of the Saber Pro exams, although the compulsory modules applied to all students in
that system focus on measuring generic competencies*. In the Colombian system,
additional discipline-specific modules are only taken by students when their higher
education institutions opt in to this component (ICFES, 2018(4}). In Mexico, CENEVAL,
an independent foundation, implements a system of discipline-specific tests for students
graduating from academic bachelor’s programmes, the Exdmenes Generales para el
Egreso de Licenciatura (EGEL). However, in the Mexican context, whether or not students
take this exam depends on the specific policies of the HEI they attend, not on specific
requirement from public authorities. Unlike Brazil, Mexico does not have a comprehensive
and compulsory system of quality assurance in higher education and EGEL is therefore
used by HEIs to monitor their own performance and demonstrate the effectiveness of their
programmes to the outside world, rather than as an input to universal evaluation and
regulatory processes (OECD, 2018 (forthcoming)}s)).

The formal objective of ENADE is set out in the 2004 legislation establishing SINAES:

ENADE shall assess students' performance in relation to the curriculum guidelines
for their respective undergraduate course, their ability to adjust to the
requirements arising from the evolution of knowledge and their competency in
understanding themes outside the specific scope of their profession, linked to
Brazilian and worldwide realities and other areas of knowledge. (Presidéncia da
Republica, 2004, p. art.5;1))

The most recent Ministry of Education ordinance describing INEP’s evaluation activities
in higher education further specifies that ENADE should measure the “skills and
competencies” that undergraduate students have “acquired in their training”. It should do
this based on the content in the relevant National Curriculum Guidelines (Diretrizes
Curriculares Nacionais); the National Catalogue of Advanced Technology Programmes
(Catalogo Nacional de Cursos Superiores de Tecnologia); “associated norms”; and
“current legislation regulating professional practice” (MEC, 2017, p. art.41,6). The
National Curriculum Guidelines (DCN) and the National Catalogue of Advanced
Technology Programmes specify expected learning outcomes for programmes in different
fields at a general level. The formal expectation is thus that ENADE should assess the
extent to which students have acquired, through their undergraduate programme, the
learning outcomes specified in these documents. In addition, as shown in the citation above,
the 2004 legislation calls for ENADE to measure the ability of students to adjust to the
“evolution of knowledge” and the ability of students to understand wider themes outside
their immediate field of study.

Defining and measuring learning outcomes in higher education in a systematic way has
been an increasing focus in higher education practice and policy in recent decades
(CEDEFOP, 2016;7)). This trend reflects a widespread recognition that gaining a better
understanding of the extent to which students really acquire new knowledge and skills
through higher education would provide useful feedback for teachers and educational
institutions and useful information for governments and society at large on the educational
performance of higher education (OECD, 2013;s;). On a conceptual level, it is desirable to
have reliable information on learning outcomes — and not simply inputs and processes
involved in the learning process — in order to judge how effective and efficient the education
provided is in practice. However, while the explicit definition of expected learning

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 115

outcomes has become the norm across many higher education systems, final assessment of
these learning outcomes is nearly always left to individual teachers and institutions (or to
employers or professional organisations). In contrast to practice at school level, where
many OECD and partner countries use external high-school leaving examinations, attempts
at the comparable measurement of student learning outcomes in higher education, between
institutions and disciplines, have been rare (OECD, 2013))).

In Europe, for example, the systematic establishment of expected learning outcomes for
higher education programmes has been a key feature of the Bologna Process. The capacity
of programmes to deliver these learning outcomes is a strong focus of many European
quality assurance models. Nevertheless, no European quality assurance system has so far
attempted to incorporate direct assessment of learning outcomes in its processes, although
some — such as that in the Netherlands (NVAO, 2016;9)) and Sweden (UKA, 201 8,10)) —
have included qualitative assessment of selected student outputs in programme-level
reviews.

The OECD initiated the Assessment of Higher Education Learning Outcomes (AHELO)
(OECD, 2013;s)) project and the European Commission has supported a project to develop
frameworks to assess learning outcomes in different higher education disciplines
(Comparing Achievements of Learning Outcomes in Higher Education in Europe -
CALOHEE) (CALOHEE, 2018 /11)). However, neither of these initiatives has yet been fully
implemented in large-scale testing.

Despite the widely accepted potential value of assessing learning outcomes in theory, in
practice, external assessments at higher education level face substantial challenges:

e A first challenge is defining what to measure. Whereas in school education, all
students are expected to follow a common core of subjects, at least until the end of
compulsory schooling, students in higher education study subjects with a narrower,
but deeper focus. Within their chosen discipline, higher education students also
tend to specialise in certain sub-domains, especially in the later years of their
course. Accurately measuring the skills and competencies acquired by students
during their programme requires testing instruments that are adequately tailored to
the content of the programmes in question. If tests are aligned too closely to specific
programme content, they can only be applied to comparatively small numbers of
students, who have been exposed to that specific content. This makes creating a
system-wide testing system complex and costly and can restrict comparability of
results to comparatively small cohorts. If the tests are made more general, to allow
them to be applied to a larger number of students, there is a risk that the test content
is no longer sufficiently aligned to the programme content to allow causal
inferences to be made about the effects of the programme on student learning.

The challenge for test developers is, therefore, where to set the boundaries of discipline-
specific knowledge, in terms of breadth and depth, to maintain a clear link to programme
content, while creating a test that is widely applicable. In some cases, such as the
Colombian Saber Pro exam, test developers have avoided this problem by focusing
compulsory test components exclusively on generic competencies and using the same test
for students from all disciplines. The first problem with this approach is that generic
competencies are often an implicit, rather than explicit, intended learning outcome of
higher education programmes. Although higher education programmes may help students
to improve their reading skills and basic problem-solving, this might be a by-product of the
main teaching and learning activities, rather than an explicit focus. Generic competency
tests therefore set out to test competencies that higher education programmes — rightly or

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

116 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

wrongly — might not have set out to develop in their students, while ignoring discipline-
specific content and skills that are the prime focus of the course.

e Asecond key challenge is how to identify and formulate test items that can measure,
in a reasonably brief test, the kinds of knowledge and discipline-specific
competencies that students might be expected to acquire through a programme
lasting three, four or more years. Alongside the breadth of theoretical knowledge
that could be covered in a written test, lies the challenge of creating assessment
models for the practical and professional skills that are central to many higher
education programmes such as nursing, medicine, engineering or architecture.
Although it is conceivably possible to cover a wide range of topics and tasks in a
standardised test lasting several hours, a balance needs to be struck between
coverage of items and creating optimal conditions for a student to perform to the
best of their ability. Long tests are tiring, while short tests can create excessive
stress that impacts negatively on student performance.

e A final issue relates to the risk that standardised testing, even when adequately
differentiated between disciplines and types of programme, can unduly influence
the programme content and pedagogical approaches used by institutions and
teaching staff. When broad curriculum guidelines are translated into a specific test,
this necessarily entails making specific judgements about what students should
learn. When the test in question matters, this encourages teachers to “teach to the
test”, with a clear risk that legitimate learning objectives and subjects not covered
by the test are excluded and that programme coordinators feel constrained in make
changes to their curricula. Standardised testing might thus constrain curriculum
breadth, innovation and responsiveness to changing circumstances.

In light of these considerations, the OECD review team considers that the objectives of
ENADE, as currently formulated, are unrealistic. The relevant legislation, noted above,
requires ENADE to measure students’ performance in relation to the content of relevant
national curriculum guidelines, their ability to analyse new information and their wider
understanding of themes outside the scope of their programme. This creates problems, as
follows:

e The requirement to measure understanding of unspecified “themes outside the
scope of the programme” is inherently problematic because it is so general and the
knowledge and skills assessed, by definition, are not part of the programme’s core
intended learning outcomes. It is thus unclear how those running programmes could
be expected to equip students with such a wider range of unspecified knowledge
and skills or why they should be held accountable for students’ not having these
competencies at the end of their studies.

e As ENADE is a written examination with a restricted duration, it is impossible to
measure the full range of learning outcomes that any adequately formulated
curriculum guidelines should contain. At best, written examinations (computer or
paper-based) can measure a scientifically robust selection of discipline-specific
knowledge and skills and, potentially, generic competencies such as logical
reasoning or problem-solving (if such competencies are intended learning
outcomes). An exam like ENADE can only ever assess a small proportion of what
students will have been expected to learn over the duration of their programme. It
would be beneficial if this were acknowledged in the stated objectives of the exam.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 117

e By implying that ENADE sets out to measure students’ learning outcomes in
relation to the National Curriculum Guidelines for undergraduate programmes,
there is a risk that the content of ENADE (in a given year or previous years) comes
to be seen to define what is important in the National Curriculum Guidelines. In
practice, the National Curriculum Guidelines in Brazil (MEC, 2018 ;12}) specify very
general content for programmes, leaving considerable scope for HEIs and teachers
to innovate and adapt content to current needs. If ENADE has an excessively
narrow approach, it risks undermining this freedom. In interviews, representatives
of HEIs in Brazil, indicated that the content of ENADE did indeed influence the
content of their programmes.

If ENADE is to be maintained, Brazil’s legislators and quality assurance authorities need
to provide a more credible account of what it can realistically achieve and how risks for
innovation and responsiveness can be mitigated.

Effectiveness: quality indicators used and generated

ENADE aims to measure student performance through an assessment composed of ten
questions focused on general competencies (formacao geral), of which two call for short
discursive answers and the rest multiple choice answers, and 30 discipline-specific
questions, of which three are discursive and 27 are multiple choice. The general
competency questions are common to all ENADE exams in a single year. Exams are taken
with paper and pen in test centres hosted by students’ higher education institutions and
students have up to four hours to complete the whole exercise. Participation in the
examination is compulsory for all students graduating in courses in the fields being
assessed in a given year, with attendance being a prerequisite for receiving the diploma for
their degree*. The tests are administered on the three-year cycle described above, with fields
following the order set out in Table 5.1.

Table 5.1. ENADE cycle

Year Fields examined
a) Bachelor's degrees in the areas of Health, Agrarian Sciences and related areas.
b) Bachelor's degree in Engineering.
1 c) Bachelor's degrees in the area of Architecture and Urbanism.

d) Advanced Technology Programmes (Tecnologias) in the areas of Environment and Health, Food Production.
Natural Resources, Military and Security.

a) Bachelor's degrees in the areas of Maths, Natural Sciences, Computing and related areas.
b) Evaluation areas leading to a double qualification with bachelor's degree and Licenciatura.
2 c) Evaluation areas leading to a Licenciatura.

d) Advanced Technology Programmes (Tecnologias) in the areas of Control and Industrial Processes, Information
and Communication, Infrastructure, Industrial Production.

a) Bachelor's degrees in the areas of Applied Social Sciences, Human Sciences and related areas.

3 b) Advanced Technology Programmes (Tecnologias) in the areas of Management and Business, School Support,
Hospitality and Leisure, Cultural Production and Design.

Source: Adapted from Regulatory Ordinance no 19, of 13 December 2017 (MEC, 2017,6}).

Both the general competencies and discipline-specific sections of ENADE are marked out
of 100, generating “raw” marks out of 100 for each section, for each participating student.
These are combined, with general competencies accorded a weighting of 25% and the
discipline-specific section 75%, to generate an overall average “raw” mark per student and
per programme. Data from the 2016 round of ENADE (INEP, 2017 113;) show that average

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

118 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

marks on the general competencies section, which is the same for all students, ranged from
38.2% for students from Advanced Technology Programmes in cosmetology to 60.3% for
(bachelor’s degrees in) medicine. Average marks for the discipline-specific components,
which are not comparable between disciplines, ranged from 38.3% in physiotherapy
bachelor’s programmes to 67% in medicine.

The raw average values for each programme in a single field are then standardised by
calculating their standard deviation (distance from the mean score for programmes in a
given field) and attributing the standard deviations to value bands from one to five, as
shown in Figure 5.2 (INEP, 2017j14}). Outliers are excluded in setting the minimum and
maximum scores. This process is carried out for all programmes with at least two
graduating students participating in the exam. The resulting score from one to five for each
programme is the ENADE score or Conceito ENADE. 40% of programmes evaluated in
2016 received an ENADE score of three and 6% a score of five (INEP, 2017113).

Figure 5.2. Standardisation of ENADE scores

Source: INEP (2017) Presentation ‘Enade 2016 Resultados e Indicadores — DAES, setembro 2017’. (INEP,
2017[13}).

Programme/HE|

Value band

The administration of ENADE is a complex exercise. Every year, INEP coordinates the
development of tests that include 30 discipline-specific items for numerous fields (27 in
2018°). The agency oversees the implementation of these tests on a single day for around
200 000 students from over 4 000 programmes in | 000 HEIs across a vast country; and
coordinates the marking of the exam papers, attribution of scores and calculation of
resulting indicators.

However, since its inception in 2004, ENADE has been subject to criticism, especially from
within the Brazilian academic community. Based on the evidence gathered during the
review, the OECD review team considers that there are at least five principal weaknesses
in the way ENADE is currently designed and implemented.

1. The first problem relates to the participation of students and their motivation to
make an effort in the test. It is evident that a proportion of the students who should
be taking the test each year are not doing so. Across years, between 10-15% of
students registered to take the test each year do not turn up on the day. Moreover,
there are concerns among stakeholders in Brazil that some HEIs seek to avoid

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 119

registering a proportion of students for ENADE. This may either be through
maintaining deliberate ambiguity about the attribution of programmes to an
ENADE test field — a problem recently highlighted by Brazil’s Federal Court of
Accounts (TCU, 2018;15)). Alternatively, it may occur as a result of deliberate
policies by HEIs to ensure the academic progress of “weak” students is slowed, so
that they are not scheduled to graduate in a year when their discipline is subject to
ENADE and thus avoid taking the exam.

At the same time, ENADE is a high stakes exam for HEIs (as it is used in the quality rating
of their programmes), but a low stakes exam for students. Although attendance is
compulsory, ENADE scores have no effect on students’ academic record and there is
evidence that a significant proportion of students do not complete large parts of the test
(Melguizo and Wainer, 2016;16}). Although some institutions organise prizes for ENADE
participation, there is no systematic use of incentives for success in the exam. Evidence
from other OECD and partner countries suggests that if the results of tests have no real
consequences for students, this impacts negatively on student motivation and performance
(Wolf and Smith, 199517; Finney etal., 2016;18)). Moreover, interviews with
representatives of HEIs conducted during the review visit to Brazil suggest that ENADE is
taken more seriously by both students and staff in less prestigious for-profit institutions,
than in public institutions. This appears to be primarily because the results of ENADE have
a greater potential impact on the reputation of private institutions, which have to compete
for students (public institutions are free and thus usually oversubscribed). This, and the
participation issues, have negative implications for the validity of the results as an accurate
reflection of the learning outcomes of all students and for the comparability of results
between programmes and institutions.

2. A second concern relates to the development, selection and use of test items for
each ENADE test. At present, there is no robust methodology to ensure that the
difficulty of each test item is taken into account in the composition of the test and
thus that a) tests in the same field are of equivalent difficulty between ENADE
cycles and b) tests in different fields are of a broadly similar level of complexity.
This means that it is not possible to compare the raw results or the Conceito ENADE
between years or between disciplines. Unlike in external school-leaving exams
across the world, no reliable attempt is made to ensure exams are of consistent
difficulty between years or that an exam in, say, philosophy is of broadly equivalent
complexity in relation to expected learning outcomes to ones in maths or chemistry.
Some differentiation in expected standards is important. For example, a bachelor’s
degree would be expected to take students to a higher level than an Advanced
Technology Programme in a related subject. However, it is reasonable to expect all
ENADE exams for equivalent levels (bachelor’s degrees, short-cycle degrees) to
test students at an equivalent level in relation to the expected learning outcomes for
their programme.

A related question is whether the number of discipline-specific items included in ENADE
(30) is adequate to generate a reliable indication of students’ learning outcomes from an
undergraduate programme. The answer almost certainly depends on what the exam is for.
A robustly designed examination with only 30 items, may be able to provide a general
indication of a students’ level of knowledge and competencies in a specific disciplinary
field. However, such a test is unlikely to provide reliable evidence of students’ performance
in specific sub-fields or aspects of the curriculum, which limits its usefulness as a tool to
help teaching staff and institutions improve the design of their programmes.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

120 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

3. A third problem is that there are no explicit quality thresholds or expected minimum
levels of performance for ENADE tests. “Raw” scores (out of 100) are widely
published and discussed, with a frequently expressed assumption that students are
not performing well (EXAME, 2017119)). However, the absence of a clearly
calibrated level of difficulty in tests means it is impossible to say whether a score
of 50% represents good or bad performance. If the test is set at a high level of
difficulty, it might be a very good mark, if the test is at a low level, it would not be
a good mark. Without tests of a comparable standard of difficulty and without
defined quality thresholds (pass, good, excellent, etc.), ENADE scores are simply
numbers. It is impossible to know if students in programmes that achieve 50% or
60% in ENADE are performing well or poorly.

4. A fourth problem relates specifically to the design of the general competencies
(formagdo geral) component of ENADE. This is currently composed of general
knowledge questions regarding current affairs and social issues, including two
questions that call for short discursive answers. This reflects the requirement of the
2004 legislation that ENADE test the wider knowledge and understanding of
students of issues outside the scope of their studies. However, as noted, this is a
flawed objective. Unless all undergraduate programmes have knowledge of current
affairs and social issues as explicit intended learning outcomes — which is not the
case — it is unreasonable to judge individual programmes on students’ performance
in these areas. It noteworthy that ENADE does not include questions designed to
test students’ logical reasoning or similar generic competencies that one might
reasonably expect all higher education graduates to possess.

5. A final issue is that the standardisation of ENADE scores compounds the lack of
transparency about what ENADE results really mean. As noted, raw marks are
simply attributed to a five-point scale based on the standard distribution of scores
in a single subject in a given year. As tests may vary in difficulty and students
obtain very different distributions of scores, where a programme falls on a standard
distribution of the scores for all programmes says little about the actual quality of
the programme in question. In simple terms, scoring a three in one discipline in a
particular year may require a far higher level of performance than scoring a three
in another discipline in the same year or the same discipline in another round of
ENADE. If all students score low marks — and thus all programmes are performing
poorly - the distribution will be skewed towards the lower end of the performance
scale, but as the bar (average) is set low, programmes will still emerge as having
scores of three and more.

Effectiveness: use and effects of ENADE results

The ENADE score is published separately for each programme in the e-MEC online
platform (MEC, 2018,)20}) and subsequently used as an input to the Preliminary Course
Score (CPC), discussed in more detail below. The CPC is used by SERES to make
regulatory decisions on whether programmes should have their official recognition
renewed directly every three years, or should undergo a further on-site inspection (MEC,
2017,2;). ENADE results are thus used by Brazilian authorities to judge whether
programmes are of adequate quality to allow their continued operation. The results of
ENADE are also widely reported in Brazilian media and, in some cases, used by institutions
as part of their marketing efforts, alongside the programme and institutional quality scores.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 121

Students have access to their own ENADE results, but it is unclear to what extent they
actually have an interest in these, given the low importance of the test for their own careers.
Importantly, the OECD team understands that institutions, and staff in programmes, do not
have access to the individual scores for their programmes. Institutions consulted by the
OECD review team report that they did not use of ENADE results in efforts to improve the
design and content of programmes. During the review visit, representatives of institutions
consistently indicated that they did not see ENADE as providing useful feedback to help
them improve their programmes. This is a pattern confirmed by recent studies on the use
of ENADE results in HEIs in Brazil (De Sousa and De Sousa, 2012)21;; Oliveira et al.,
2013,22;; Santos et al., 2016;23)). In contrast, some HEI representatives consulted by the
OECD team, in line with the findings of the same studies, highlighted the use of ENADE
results in marketing programmes to prospective students and promoting their institutions.

Efficiency: the cost-effectiveness of ENADE

Although the OECD review team does not have access to a detailed breakdown of the costs
of implementing ENADE, these account for a substantial part of INEP’s budget for
evaluation of higher education, which amounted to over 118 million reals (USD 30.7
million) in 2017 (INEP, 2018)24)). It is questionable whether the quality and usefulness of
the results achieved with the exam as currently configured justify the investment of public
resources committed. As noted, through their use in the CPC, ENADE results are used by
MEC as an important indicator of the quality of undergraduate programmes in Brazil.

The first question is therefore whether ENADE provides an accurate indicator of
programme quality. For the reasons outlined above, it is the view of the OECD review team
that, in its current form, it does not. The second question is whether an improved version
of ENADE, addressing the current design and implementation weaknesses noted in the
preceding section, would be generate information about the quality of undergraduate
programmes that could not be provided by other, potentially more readily available,
indicators. The third question is whether, even if an improved ENADE brought additional
information about quality that could not be provided by other indicators, this additional
information justified the considerable costs of designing and implementing such a system
of standardised testing. These are questions that the Brazilian authorities and academic
community should consider in planning the future of the federal quality assurance and
regulatory regime for higher education.

Monitoring programme performance: use of programme indicators

Relevance: rationale and objectives of the current system

INEP and MEC use a set of programme-level indicators, drawing heavily on ENADE
results, to monitor the ongoing performance of undergraduate programmes in the federal
system of higher education. The legal basis for this is, in principle, provided for in the 2004
legislation establishing SINAES, which states that evaluation of programmes “shall use
diversified procedures and instruments, among which must be visits by expert committees
from the respective areas of knowledge” (Presidéncia da Republica, 2004, p. art.4;1))”. On-
site reviews are always used as a basis for initial recognition of undergraduate programmes,
but are not used systematically in the ongoing monitoring of programme performance and
the periodic renewal of programme recognition. As discussed below, in many cases,
SERES makes judgements about the quality of programmes and approves the renewal of
their official recognition based on programme indicators generated by INEP.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

122 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

A recent MEC implementation ordinance for SINAES (MEC, 2017{6}) states that INEP has
the responsibility to calculate and publish indicators of the quality of higher education, in
line with methods established in technical notes approved by CONAES (the supervisory
board for the higher education evaluation system). It does not, however, specify in detail
what these indicators should be. The same ordinance states simply that indicators produced
by INEP are to “support” (subsidiar) public policies in higher education. In practice,
however, centrally collated programme indicators play a central role in the federal quality
assurance system. The current indicators and methods used, as well as their application in
regulation and supervision by SERES, are specified in technical notes prepared by MEC
and INEP (MEC, 2017)2); INEP, 2017,22)).

In the current system, INEP and MEC use indicators as a means to identify potentially poor
quality programmes that warrant more in-depth evaluation and supervision. Programmes
that score below three out of five on the composite Preliminary Course Score (CPC) after
each three-year ENADE cycle are systematically subject to on-site inspections by external
review commissions, with a positive evaluation score (a CC of three or above) a
prerequisite for renewal of their official recognition. Courses that score three or above on
the CPC generally have their programme recognition renewed automatically by SERES
(MEC, 2017))).

In a system as large as Brazil’s, with a wide diversity in the quality of provision, there is a
clear rationale for using centrally collated indicators to monitor programme quality.
Notwithstanding the minimum quality guarantees provided by the recognition process,
wider quality concerns in the higher education system in Brazil, particularly in parts of the
for-profit private sector, mean that ongoing monitoring of programmes by public
authorities is justified. However, in a system of such size, it is challenging to conduct
regular programme-level reviews for all active undergraduate programmes, as is the
practice in some smaller OECD higher education systems (such as the Netherlands or
Portugal, for example).

Despite the practicality of an indicator-based approach, some commentators in Brazil
consulted by the OECD review team question whether dispensing with on-site reviews for
some programmes in favour of indicator measures is consistent with the spirit of the 2004
legislation (which says programme evaluation must involve on-site visits). More
fundamentally, the question is which indicators to use to monitor programme quality and
how to combine these in meaningful ways.

Effectiveness: quality indicators used or generated

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 123

Table 5.2. Indicators used to calculate the CPC

Dimension Components Weights
Score from students taking ENADE 20.0%
Student performance Indicator of Difference between Observed and Expected 35.0% 55%
Performance (IDD) ;
Proportion with master’s 7.5%
Staff (Corpo Docente) Proportion with PhDs 15.0% 30%
Score for employment status of staff (regime de trabalho) 7.5%
Score relating to the organisation of teaching and learning 7.5%
Student perception about Score relating to physical infrastructure 5.0% 15%
the educational process Score relating to additional academic and professional training 9 5%
opportunities :

Source: INEP (2017) Nota Técnica N° 3/2017/CGCQES/DAES (INEP, 2017,22)).

Under “student performance”, the ENADE score (Conceito ENADE) discussed in the
previous section is complemented by an indicator of the assumed “added value” of the
programme for each student’s performance in ENADE. The Indicator of difference
between observed and expected performance (/ndicador da Diferenga entre os
Desempenhos Observado e Esperado) or IDD is calculated by comparing each student’s
actual results in ENADE with the performance that would be expected on the basis of their
previous performance in the national high-school leaving exam, ENEM (Exame Nacional
do Ensino Médio).

This process uses results data associated with each individual’s unique identification
number (numero do Cadastro de Pessoas Fisicas or CPF). The “expected” performance
for each student in ENADE in relation to the total population of ENADE participants is
calculated on the basis of ENEM results using a regression model. This is then compared
with the student’s actual performance in ENADE. The difference between the expected and
observed performance is considered as the added value of the programme (positive or
negative). The average differences for all students® in a single programme are combined to
create an IDD score for the programme (INEP, 2017;23)).

The data on the composition of staff are drawn from INEP databases and awards higher
ratings to programmes with higher proportions of teaching staff with master’s and doctoral
degrees and with full-time, permanent status. This tends to reward public institution that
have higher proportions of staff with doctorates (see Figure 3.7) and full-time contracts
(see Table 3.3).

The data on student feedback are drawn directly from the online student questionnaire that
students have to complete as part of their registration for ENADE. The 2017 questionnaire
contained 26 questions asking for biographical details and a further 41 questions focused
on the programme (INEP, 2018)24]). Raw data for the student questionnaire results and staff
composition are converted into standardised scores for each programme.

The CPC has been criticised in Brazil for several reasons. Criticisms focus on the choice
of indicators, the weighting attributed to them and the way they are combined into a single
composite indicator. The reliability of the ENADE score as an indicator of programme
quality was discussed above. The IDD, which accounts for 35% of the CPC score, is both
contested and poorly understood in the wider academic and policy community in Brazil.

The objective of measuring the added value of a programme for the knowledge and skills
of students (learning gain) is, in principle, commendable. Not only does it go to the heart

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

124 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

of educational effectiveness (how much a student actually learns during their programme),
but measuring “added value” also makes it possible to take into account and reward the
efforts made by programmes that take in weaker students, often from less advantaged social
groups. A measurement of learning gain can, in principle, make it possible to identify
programmes that successfully help students with lower levels of knowledge and skills on
entry to increase their knowledge and skills over the duration of the programme, even if the
final performance of these students remains lower than in programmes that took in high
performing students. Added value could thus be a means to recognise the work of less
prestigious programmes and institutions that perform a valuable societal role.

However, the IDD as currently calculated in Brazil is not a true measure of learning gain,
but a proxy indicator, based on the following assumptions:

1. That students’ performance at the age of 18 in general, high-school-level tests in
natural sciences, humanities, languages and maths (INEP, 2017(23)) is a reliable
basis on which to predict their future performance in tests designed to measure their
learning outcomes after a specialised undergraduate degree.

2. That ENADE accurately measures students’ performance at the end of their
undergraduate programme.

3. That when students perform better or worse than their predicted relative
performance, this results primarily or significantly from the design and delivery of
their undergraduate programmes.

The first assumption is not unreasonable. However, the limitations of ENEM results as a
predictive indicator must be fully acknowledged and the results of the regression analysis
(the predicted relative performance of students at the end of their undergraduate studies)
must be treated with due caution. The subject composition and different stakes involved in
ENEM and ENADE are likely to reduce the predictive power of the IDD calculations. Not
only will ENEM subjects differ considerably in scope and depth from the subjects included
in the disciplinary component of ENADE, but, as highlighted earlier, ENADE is a low
stakes exam for students. In contrast, ENEM is comparatively high stakes exam, the result
of which have a major impact on students’ chances of accessing higher education.
Depending on their profile, the relatively high degree of specialisation of higher education
programmes might increase some students’ chances of performing well, compared to
generalist high-school exams (assuming they chose to study fields in higher education that
correspond to their strengths and interests). Conversely, the low stakes of ENADE tests
may reduce the chance of their performing to their maximum potential.

The second assumption is problematic. As discussed above, as well as their low-stakes
status, the composition and design of ENADE tests limit their effectiveness as measures of
learning outcomes.

These factors also reduce the reliability of the third assumption above. All quality assurance
procedures make assumptions about the impact of programme design and delivery on
students’ performance. It is certainly possible that programmes will have some influence
on students’ performance in the current ENADE tests and that significant differences
between real and expected performance, as calculated by the IDD, are due to programme-
level factors. However, the combination of the boldness of the underlying assumptions, the
poor design of ENADE and the potential influence of factors outside the control of the
programme on student performance mean that the IDD provides only limited information
on programme quality. It is thus highly questionable whether it should account for 35% of
the weight in a composite indicator used as a basis for renewing programme recognition.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 125

The legislation establishing ENADE actually calls for the testing of both first and final year
students, in order to allow a direct comparison between their results on entry to, and exit
from, undergraduate programmes. This system was implemented between 2004 and 2010.
However, the test for incoming students was abandoned, largely on cost grounds, and the
current system using ENEM introduced as a substitute. Although testing on entry and exit
from a course with coordinated and well-calibrated testing tools also involves many
challenges, the OECD team considers it is the only realistic way to measure learning gain
within programmes, if this is the real objective of quality assurance authorities.

The indicators used for the composition of teaching staff and student feedback are less
ambitious in their aims and less fundamentally problematic. However, by rewarding staff
with PhDs and full-time status, the indicators for teaching staff are designed for academic,
research-oriented institutions and fail to take into account the value of part-time teachers
with professional experience, who are vital for more professionally oriented programmes.

The use of student feedback as an indicator is a positive element in the system. Students
are the main stakeholders in the higher education process and their views need to be heard
within a well-designed quality assurance system. However, the questionnaire used is long
and formulated using excessively bureaucratic language (INEP, 201824). Moreover, it is
not clear whether students in Brazil, who sit the ENADE tests in their own institutions, are
accustomed to providing honest and objective feedback about their teachers and
programmes throughout their studies and whether they are positively encouraged to do so
in ENADE. In particular, a concern to maintain the reputation of the institution that has
awarded their diploma may make students hesitant to provide negative feedback.

It is widely accepted, including in the recent report from the Federal Court of Accounts
(TCU, 2018115)), that the weightings attributed to the different indicators in the CPC are
arbitrary, with no discernible scientific basis. This further compounds the lack of
transparency for students, families and society at large about what the scores attributed to
courses really mean in practice. It is possible — although the OECD review team is not
aware of specific studies — that the current weightings in the CPC formula have a significant
impact on where programmes fall in standard distributions and thus their CPC score.

It is positive that the CPC sets out to include indicators of the teaching process (through
the imperfect proxy of teaching staff status); qualitative feedback from students (the main
beneficiaries of the system) and measures of student learning outcomes. It does not,
however, contain a measure of the attrition rate of students (what proportion of students
entering a programme complete it) or the subsequent employment outcomes of graduates.
Both these factors — although especially the first - are taken into account in quality-related
policies in other higher education systems.

Effectiveness: use and effects, efficiency and cost-effectiveness

In light of the shortcomings of the set of indicators employed and the arbitrary weighting
of the different factors, a key concern is the way the CPC is currently used in the broader
quality assurance and regulatory process for undergraduate provision. As discussed,
programmes that score three or above, in most cases, have their recognition renewed
automatically until the next ENADE cycle, at which point a new CPC is calculated’. Those
that score two or less are systematically subject to a review visit (discussed below), which
provides an updated programme score (CC) as a basis for the regulatory decision on
whether or not to renew recognition.

RETHINKING QUALITY ASSURANCE FOR HIGHER EDUCATION IN BRAZIL © OECD 2018

126 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

As argued earlier, the principle of using indicators to identify “at risk” programmes and
target scarce resources for on-site inspections makes sense, especially in a system as big as
Brazil’s. However, the CPC does not provide a reliable mechanism to identify poorly
performing courses. The absence of quality thresholds in ENADE and the standardisation
processes used to create the ENADE score, combined with the weaknesses of the IDD,
mean it is far from clear whether a CPC score of three represents an adequate standard of
quality or not. A reform of the monitoring indicators used and the way they are combined
is necessary.

Monitoring programme performance: use of on-site inspections

Relevance: rationale and objectives of the current system

When programmes are identified through the CPC as performing poorly — often meaning
they have poor relative performance in ENADE — they are subject to an on-site inspection
by external evaluators, coordinated by INEP (MEC, 2017, p. 5,2}). The evaluators assess
the supply conditions for the programme using the same evaluation template that was
already used for programme recognition (reconhecimento) and was discussed in Chapter 4
of this report (INEP, 2017;)3]). The results of the new on-site inspections triggered by the
CPC process or special requests from SERES are used as a basis for decisions for
programmes’ renewal of recognition.

As noted, the objective of targeting on-site inspections on weakly performing programmes
has advantages, as the systematic use of periodic on-site inspections for all programmes in
Brazil would almost certainly be unfeasible for logistical and financial reasons. However,
it also means programme-level site visits at this stage in the evaluative process always have
a punitive character and that peer reviewers are not exposed to good practice in well-
established programmes, which could inform their judgements about, and
recommendations to, poorly performing programmes.

The objective of the on-site visits for renewal of recognition, as currently conceived, is to
(re)check compliance with basic standards, rather than explicitly to promote and support
improvement of the programmes concemed, following a serious self-evaluation. The
formal objective for the renewal of recognition visits expressed in the relevant regulation
(Presidéncia da Republica, 2017,25)) is for them to inform the process of renewal of
recognition. The extent to which quality assurance should seek to support institutions in
quality improvement is open to debate in a highly commercialised system like that in
Brazil, although most quality assurance systems worldwide do seek to support quality
enhancement, as well as basic compliance.

Effectiveness: quality indicators used and generated

The indicators used in the evaluation template for recognition and renewal of recognition
of programmes were discussed in Chapter 4. A clear problem with these indicators in
relation to renewal of recognition is that they focus on exactly the same inputs and
processes as examined in the initial recognition of the programme, when the first cohort of
students had only completed between 50-75% of the course workload, and before any
students had graduated. The courses that undergo renewal of recognition on-site inspections
are identified primarily because of their poor performance on an — albeit imperfect — set of
indicators of output and student feedback. Rather than directly examining the problems
identified by poor performance in ENADE or poor student feedback, the on-site evaluators
complete a questionnaire measuring variables that have already been verified in an earlier

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 127

inspection. Moreover, these variables in many cases might be expected to remain constant
over time.

It should be recalled that the evaluation template for recognition and renewal of recognition
places a 40% weighting on the category “teaching staff’ and 30% on “infrastructure”, with
the remaining 30% on teaching and learning policies and practices. The indicators and
judgement criteria relating to teaching staff mostly focus on the qualifications and
experience of the individuals in question, with only three indicators dealing with the
activities (atuagao) of staff or their interaction with each other.

It is conceivable that some, very poor quality, providers do not maintain the basic
infrastructure and teaching workforce to allow their programme to operate correctly and
that were initially verified in the recognition process. However, for programmes that have
maintained the basic conditions for providing the programme, the balance of indicators in
the current evaluation template does not generate overall scores that will reveal the most
obvious signs of poor quality. A greater focus on teaching activities and outputs and
outcomes (attrition rates, learning outcomes, graduation rates and employment outcomes),
would make the evaluation template more effective in identifying the real causes of poor
performance.

Effectiveness: use and effects, efficiency and cost-effectiveness

Programmes are inspected at this stage in the evaluation cycle on the grounds that they
have scored poorly on the CPC measures of process and outcomes. The resulting inspection
then attributes a new quality score — an updated Conceito de Curso (CC) — that effectively
replaces the CC attributed at the time of initial recognition and exists alongside the CPC
score in the e-MEC system. The CC is, however, based on entirely different indicators from
the CPC, despite the very similar names.

Frequently, it appears that programmes which score poorly on the CPC measure
subsequently achieve a higher score on the CC (TCU, 2018r15)). As such, these programmes
nominally recover the higher quality score. It is entirely understandable that the CPC and
the inspections leading to the CC can generate different values. They measure almost
entirely different things.

From a conceptual perspective, this is all the more problematic because the more output-
focused measures contained in the CPC would — if calculated on a more reliable basis —
provide a better indication of the real performance of the programme. Curriculum plans,
teachers and infrastructure are all enabling factors for good education and are rightly
considered in initial programme approval. However, notwithstanding the potential for staff
to leave the programme and infrastructure to be changed, these factors are likely to remain
constant over a number of years. Once there is evidence of the broader performance and
outcomes of programmes, this evidence should be prioritised in assessments of quality. In
the end, if a well-designed programme with good infrastructure and well-qualified teachers
fails to train students effectively without very good reasons, the programme is not of high
quality.

128 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

5.3. Key recommendations

1. Undertake a thorough assessment of the objectives, costs and benefits of
large-scale student testing as part of the quality assurance system

Given the long-standing commitment of Brazil’s public authorities to standardised testing
of tertiary students, there are sunk costs and considerable expertise in implementing large-
scale testing has been developed. Politically, ENADE is widely accepted and viewed by
many as an important part of Brazil’s system for quality assurance in higher education.

The OECD team believes, however, that, in its current form, ENADE does not represent
an effective use of public resources. As such, as a basis for decisions on the future of the
system, a thorough reflection is needed about the objectives of large-scale standardised
testing in Brazilian higher education and the costs and benefits of different approaches to
implementing it. The main questions to answer are:

The OECD team believes two factors should be considered in particular. First, for ENADE
to have the greatest possible added value, it needs to be able to provide reliable information
that can help teachers and institutions to identify areas of weakness in their programmes
(in terms of knowledge coverage or skills development). ENADE results cannot simply be
a blunt indicator used to inform the regulatory process, as other indicators such as attrition
rates or employment outcomes could be used for this purpose. Second, the current
requirement to apply the ENADE test to all programmes every three years increases the
fixed cost of implementing the system. It is important to consider whether sampling
techniques could be deployed to reduce costs, while maintaining reliability.

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 129

2. If a reformed version of ENADE is retained, ensure the objectives set for the
exam are more realistic

e Data on student performance that can be used directly by teachers and institutions
in identifying weaknesses in their programmes as a basis for improvement (quality
enhancement).

3. Improve the design of ENADE tests to ensure they generate more reliable
information on learning outcomes that can also be used by teachers and HEIs

As part of this process, performance thresholds and grades should be established clearly in
advance. The objective to be to provide students with easily understood grades and
programmes with easily understood and usable grade point averages and grade
distributions. The approaches to both test design and performance thresholds used by
CENEVAL in Mexico or testing organisations in the United States could provide valuable
inspiration on how a revised form of the ENADE tests could be developed. It is important
for INEP to draw on the expertise of other organisations involved in standardised testing
internationally in the development of new approaches and test formats, to ensure it benefits
from a wide range of expertise.

4. Explore ways to make the results of ENADE matter for students

130 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

At the least, the ENADE score could be included in the students’ diploma supplement.
Alternatively, ENADE could be made into a curriculum component for the years in which,
or — in the case of sampling - for the students to whom, it is administered with the
requirement that an equivalent test for students in other years be administered by
institutions. It is not yet clear if this would be possible legally.

5. Introduce a new indicator dashboard, with a broader range of measures, to
monitor programme performance and identify “at risk” programmes

The use of the Preliminary Course Score (CPC) cannot be justified in its current form for
the reasons discussed above. However, systematic programme-level data is a crucial tool
for monitoring a system as diverse and variable in quality as Brazil’s. The most promising
option would be to include a broader set of more transparent indicators in an ongoing
monitoring system, with thresholds established to indicate “at risk” performances on
different indicators. This information could then be used to inform regulatory decisions and
feed into subsequent evaluation steps (such as on-site reviews). The system should apply
to all programmes, with data obtained from institutions and other sources, as appropriate,
and consolidated in a renewed version of e-MEC.

Such a system could use a more diverse set of indicators of teaching staff, real (not
standardised) ENADE results (based on established performance thresholds), an indicator
of drop-out rates and, when possible through linking data sources through the CPF number,
information on employment rates and earnings. Indicators of the socio-economic profile of
students could be included in the system, with higher tolerances for issues like drop-out or
ENADE performance for programmes with intakes from lower socio-economic groups.
Such variation in tolerances should be limited, as all students should be expected to reach
minimum standards and all programmes maintain a certain proportion of their students.

A revised form of the IDD could potentially be maintained alongside the other indicators
in the indicator dashboard, provided its status as a proxy for expected performance and its
limitations are made clear, and its weight in the overall monitoring system is reduced.
However, even a revised IDD is likely to remain a complex indicator that may not always
be well understood by the public and stakeholders in the higher education system. The costs
and benefits of maintaining such a comparatively un-transparent measure should be
assessed. If, in the longer term, resources permit, a return to examination on entry and exit
from programmes could be considered, although the costs and benefits of any such system
should be considered carefully.

6. As part of a new system of institutional accreditation, exclude institutions
with demonstrated internal quality assurance capacity from on-site programme
reviews for the duration of their accreditation period

As discussed in Chapter 7 there is scope to exempt institutions from systematic programme-
level review that have a track record of good performance and that can demonstrate a high
level of internal quality assurance capacity. This would require existing systems for
institutional accreditation and re-accreditation to be strengthened. If problems are identified
in relation to programme indicators in the indicator dashboard, in the first instance, such

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 131

institutions would be responsible for addressing these issues internally. Addressing poor
quality would become a key focus of institutional review and poor performance or failure
adequately to address problems could lead institutions losing self-accrediting status in the
subsequent round of institutional review. This move would reduce some of the burden of
external programme-level reviews for renewal of recognition (as well as the initial
recognition process).

7. Maintain programme-level supervision for other institutions, with targeted
on-site reviews for poorly performing programmes and randomly selected
highly performing programmes.

Problematic programmes could first be called upon to submit an improvement plan that
could be assessed remotely, largely in line with current supervision procedures. SERES, or
a future quality assurance agency (see below), could decide on timeframes for improvement
and whether and when an on-site visit would be required. It is crucial that SERES, or a
successor agency, have the capacity to close poor programmes rapidly if programme
indicators fail to improve without clear justification and evaluators give a negative
assessment following an on-site inspection.

8. Develop a separate evaluation instrument for on-site reviews of established
programmes

132 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

Notes

' Every three years is the legal minimum frequency, established by the 2004 legislation, for applying
ENADE to every programme.

> The CPC is not “preliminary”, as each programme already has another score — the Conceito de
Curso (CC) — resulting from the initial recognition process that all programmes must undergo.
Moreover, when programmes are not subject to further on-site reviews, the CPC is not replaced by
a “final” score, but rather exists alongside the initial CC awarded to the programme.

3 Critical reading; quantitative reasoning; citizen competencies; written communication; and
2 b] >
English.

4 The relevant regulations provide for a limited number of acceptable reasons for being excused
from ENADE.

> 14 at bachelor’s level and 13 Advanced Technology Programmes.

° In order for a programme to have the IDD calculated, it must have a minimum of two graduating
students in ENADE with information in the ENEM database dating from the previous three years.
It also needs 20% of the total number of participating ENADE students to have data in the ENEM
database.

7 Point III.5 of the latest INEP Technical Note on the use of evaluation in the regulatory process
states: “Other already recognised courses that have achieved a satisfactory result (CPC = 3) in the
CPC of the reference year 2015, which are not included in the situations described in the previous
paragraphs: The recognition renewal process shall be opened by the Ministry of Education and the
act shall be issued in sequence without the need for a manifestation by the HEI and without any
[additional] formality.” (MEC, 2017;70)).

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES | 133

References

CALOHEE (2018), CALOHEE Outcomes Presented - CALOHEE, https://www.calohee.eu/ (11)
(accessed on 16 November 2018).

CEDEFOP (2016), Application of learning outcomes approaches across Europe: A comparative [7]
study, Publications Office, Luxembourg, http://dx.doi.org/10.2801/735711.

De Sousa, B. and J. De Sousa (2012), “Resultados do Enade na Gestaéo Académica de Cursos de [21]
Licenciaturas: Um Caso em Estudo”, Estudos em Avaliacao Educacional, Vol. 23/52,
pp. 232-253, http://publicacoes. fcc.org.br/ojs/index.php/eae/article/view/1938/1921.

EXAME (2017), Estudantes acertam menos de 50% das provas especificas do Enade (Students [19]
score less than 50% on discipline-specific ENADE tests), EXAME,
https://exame.abril.com.br/brasil/estudantes-acertam-menos-de-50-das-provas-especificas-do-
enade/ (accessed on 16 November 2018).

Finney, S. et al. (2016), “The Validity of Value-added Estimates from Low-Stakes Testing [18]
Contexts: The Impact of Change in Test-Taking Motivation and Test Consequences”,
Educational Assessment, Vol. 21/1, pp. 60-87,
http://dx.doi.org/10.1080/10627197.2015.1127753.

ICFES (2018), Informacion general del examen Saber Pro, Instituto Colombiano para la [4]
Evaluacion de la Educacion, http://www2.icfes.gov.co/estudiantes-y-padres/saber-pro-
estudiantes/informacion-general-del-examen (accessed on 13 November 2018).

INEP (2018), ENADE - Questionario do Estudante, Sistema ENADE, 27]
http://portal.inep.gov.br/questionario-do-estudante (accessed on 17 November 2018).

INEP (2018), Relatorio de Gestao do Exercicio de 2017 do Instituto Nacional de Estudos e 24]
Pesquisas Educacionais Anisio Teixeira - Inep.

INEP (2017), Enade 2016 Resultados e Indicadores, Diretoria de Avaliagaéo da Educagao 13]
Superior - Daes, Brasilia,
http://download.inep.gov.br/educacao_superior/enade/documentos/2017/apresentacao_resulta
dos_enade2016.pdf (accessed on 16 November 2018).

INEP (2017), Jnstrumento de Avaliagdo de cursos de graduacao Presencial e a distancia - [3]
Reconhecimento e Renovagao de Reconhecimento (Evaluation instrument for undergraduate
programmes - classroom-based and distance - recognition and renewal of recognition),

Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, Brasilia,
http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

INEP (2017), Nota Técnica No 3/2017/CGCQES/DAES (On the calculation of the CPC), [25]
Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, Brasilia,
http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2015/nota_tecnica_daes

n32017_ calculo_do_cpc2015.pdf (accessed on 16 November 2018).

134 | CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

INEP (2017), Nota Técnica N° 32/2017/CGCQES/DAES (Methodology for calculating the [14]
Conceito ENADE), Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira,
Brasilia,

http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_tecnica_n32
2017 cgcges daes_calculo conceito_enade.pdf (accessed on 16 November 2018).

INEP (2017), Nota Técnica N° 33/2017/CGCQES/DAES (On calculation of the IDD), Instituto [26]
Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, Brasilia,

http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_ tecnica _n33
2017 _cgcqes_daes_calculo_idd.pdf (accessed on 17 November 2018).

MEC (2018), Diretrizes Curriculares - Cursos de Graduagdao (Curriculum Guidelines -
Undergraduate Programmes), Ministério da Educagao,

http://portal.mec.gov.br/component/content/article?id=12991 (accessed on
06 November 2018).

[12]

MEC (2018), e-MEC Instituigées de Educagdo Superior e Cursos Cadastrados (Registered HEIs _ |291
and programmes), http://emec.mec.gov.br/ (accessed on 14 November 2018).

MEC (2017), Nota Técnica N° 13/2017/CGARCES/DIREG/SERES/SERES (setting out [2]
parameters and procedures for the renewal of recognition of programmes), Ministério da
Educagcao, Brasilia,

http://portal.mec.gov.br/index.php?option=com_docman&view=download&alias=6492 1 -nt-
13-2017-seres-pdf&category_slug=maio-2017-pdf&Itemid=30192 (accessed on
15 November 2018).

MEC (2017), Portaria Normativa No 19, de 13 dezembro de 2017 Dispée sobre os [6]
procedimentos de competéncia do Instituto Nacional de Estudos e Pesquisas Educacionais

Anisio Teixeira - INEP referentes a avaliacao de instituigdes de educagao superior, de cursos

de graduacdao e de desempenho académico de estudantes., Ministério da Educagao, Brasilia,
http://www.abmes.org.br (accessed on 15 November 2018).

Melguizo, T. and J. Wainer (2016), “Toward a set of measures of student learning outcomes in

[16]
higher education: evidence from Brazil”, Higher Education, Vol. 72/3, pp. 381-401,
http://dx.doi.org/10.1007/s10734-015-9963-x.

NVAO (2016), Assessment framework for the higher education accreditation system of the [9]
Netherlands, Nederlands-Vlaamse Accreditatieorganisatie (NVAO), The Hague,

https://www.nvao.com/system/files/procedures/Assessment%20Framework”20for%20the%2
OHigher%20Education%20Accreditation%20System%20o0f%20the%20Netherlands%202016
0.pdf (accessed on 17 November 2018).

OECD (2013), Assessment of Higher Education Learning Outcomes Feasibility Study Report [8]
Volume 1 - Design and implementation Executive Summary, OECD, Paris,
http://www.oecd.org/edu/ahelo (accessed on 16 November 2018).

OECD (2018 (forthcoming)), The Future of Mexican Higher Education: Promoting Quality and
Equity, http://Forthcoming.

[5]

CHAPTER 5. ASSURING AND PROMOTING QUALITY FOR EXISTING UNDERGRADUATE PROGRAMMES

| 135

Oliveira, A. et al. (2013), “Politicas de avaliacao e regulacao da educacao superior brasileira:
percepgoes de coordenadores de licenciaturas no Distrito Federal”, Avaliacdo: Revista da
Avaliagao da Educagao Superior (Campinas), Vol. 18/3, pp. 629-655,
http://dx.doi.org/10.1590/S1414-40772013000300007.

Presidéncia da Republica (2017), Decreto N° 9.235, de 15 de dezembro de 2017 - Dispoe sobre o
exercicio das funcdes de regulagdao, supervisdo e avaliagao das instituigdes de educagao
superior e dos cursos superiores de graduagdo e de pos-graduagao no sistema federal de
ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of
regulation, supervision and evaluation of higher education institutions and undergraduate
and postgraduate courses in the federal education system),
http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on
10 November 2018).

Presidéncia da Republica (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional
de Avaliagao da Educagao Superior — SINAES e da outras providéncias (Law 10 861
establishing the National System for the Evaluation of Higher Education - SINAES and other
measures), http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/110.861.htm
(accessed on 10 November 2018).

Santos, M. et al. (2016), “O impacto das avaliagées disciplinares no ensino superior”, Avaliagao:
Revista da Avaliagao da Educagdo Superior (Campinas), Vol. 21/1, pp. 247-261,
http://dx.doi.org/10.1590/S1414-40772016000100012.

SIAFI (2018), Siafi - Sistema Integrado de Administragao Financeira do Governo Federal -
Main Expenditures INEP, Siafi - Sistema Integrado de Administragao Financeira do Governo
Federal.

TCU (2018), RA 01047120170 Relatorio de auditoria operacional. Atuagao da Secretaria de
Regulagdo e Supervisdo da Educagao Superior do Ministério da Educagdao - Seres/Mec e do
Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira - Inep nos processos
de regulacdao, supervisdo e avaliagao dos cursos superiores de graduacao no pais.
(Operational audit report: Activities of SERES/MEC and INEP related to regulation,
supervision and evaluation of higher education programmes in the country), Tribunal de
Contas da Uniao, Brasilia, https://tcu.jusbrasil.com.br/jurisprudencia/5829 15007/relatorio-de-
auditoria-ra-ra-1047120170/relatorio-582915239?ref=juris-tabs (accessed on
13 November 2018).

UKA (2018), Guidelines for the evaluation of first-and second-cycle programmes,
Universitetskanslersambetet (Swedish Higher Education Authority), Stockholm,
http://www.uka.se (accessed on 17 November 2018).

Wolf, L. and J. Smith (1995), “The Consequence of Consequence: Motivation, Anxiety, and Test
Performance”, Applied Measurement in Education, Vol. 8/3, pp. 227-242,
http://dx.doi.org/10.1207/s15324818ame0803_3.

[22]

[28]

[1]

[23]

[29]

[15]

[10]

[17]

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 137

6. Assuring the quality of postgraduate education

This chapter focuses on the mechanisms used for the external quality assurance of
academic postgraduate education in Brazil. Brazilian postgraduate education comprises
“stricto sensu” programmes, with a strong academic and scientific focus, and vocationally
oriented “lato sensu” programmes, such as Masters of Business Administration (MBA).
“Stricto sensu” programmes are subject to a specific system of quality evaluation and
regulation, implemented by the Foundation for the Coordination of Improvement of Higher
Education Personnel (CAPES). The chapter analyses these processes, examining the
systems in place to evaluate new courses, to allow them to enter the National Postgraduate
System, as well as the periodic programme reviews that are undertaken every four years.
Based on the strengths and weaknesses identified, the chapter provides recommendations

for fine-tuning the system and planning for the future.

138 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

6.1. Focus of this chapter

A focus on academic postgraduate provision

This chapter focuses on the mechanisms in place at national level in Brazil to assure the
quality of postgraduate education in the country. As noted in Section 3.4, Brazilian
postgraduate education falls into two distinct categories. Courses with a strong academic
and scientific focus, which include master’s degrees (mestrado académico), Professional
master’s degrees (mestrado profissional) and doctoral education (doutorado), are classified
as “stricto sensu” postgraduate provision and form part of the National System of
Postgraduate Education (SNPG). They are subject to a specific system of quality evaluation
and regulation, implemented by the Foundation for the Coordination of Improvement of
Higher Education Personnel (Coordenagao de Aperfeigoamento de Pessoal de Nivel
Superior, CAPES), a public foundation under the responsibility of MEC. In parallel, many
higher education institutions offer professionally oriented, postgraduate “specialisation”
programmes, including Master’s in Business Administration (MBA), which are classified
as “lato sensu’ provision and are not subject to external programme-level quality assurance
as part of SINAES or organised by CAPES’. The focus in this chapter is on the quality
assurance processes for stricto sensu postgraduate provision.

Academic master’s degrees are still viewed as research degrees

The existence of a separate, long-established and highly developed system of external
quality assurance for postgraduate education programmes is a distinctive feature of
Brazilian higher education, reflecting the historical development of the science base in the
country. In many OECD countries, external quality assurance of master’s courses is
undertaken by the agencies responsible for supervision of undergraduate education, often
with the implicit understanding — in Europe notably - that master’s degrees are a vehicle to
deepen knowledge and skills gained at undergraduate level and a requirement for a wide
variety of jobs in the economy. In Brazil, in contrast, stricto sensu master’s courses —
including so-called “Professional Master’s” — are widely understood as the first stage in an
academic or research career — a situation that is largely a reflection of the relatively recent
expansion of doctoral education in the country.

A system with strong external regulation of doctoral programmes

Responsibility for the quality of doctoral training in many higher education systems
internationally has been left to individual universities, with limited intervention from public
authorities (European University Association, 2018,:))*. In such cases, incentives and
signals relating to the way programmes are organised are (increasingly) provided in an
indirect way by public research funding agencies, through the criteria used to award
doctoral training or research grants. The more direct approach adopted by public authorities
in Brazil reflects a long-standing concern to expand the population of highly qualified
researchers in the country, as a means to boost domestic scientific and innovation capacity.
In practice, as discussed below, aspects of the evaluation system implemented by CAPES
to ensure the quality of postgraduate training share characteristics with assessments
undertaken in other higher education systems to monitor the research performance of higher
education institutions.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 139

6.2. Strengths and weaknesses of the current system

CAPES evaluates and regulates market entry of new courses and oversees the
periodic evaluation of established programmes

The system of external quality assurance for academic postgraduate education in Brazil
began in its current form in 1998. It comprises two distinct processes:

1. Evaluation of proposals for new courses as a basis for regulating the entry of
postgraduate training to the system and;

2. Periodic evaluations of established postgraduate programmes, currently
undertaken on a four-year cycle, allowing their continued operation (permanéncia)
or, in case of poor performance, leading to their closure.

CAPES is responsible for coordinating the evaluation process. The evaluation of courses
and programmes is undertaken by selected academic peers from the same scientific field
working in field committees (Comissées de Area), with the scores attributed by the field
committees to proposals and existing programmes subsequently approved (or adjusted) by
the Technical and Scientific Council for Higher Education (Conselho Técnico e Cientifico
da Educagao Superior - CTC-ES) composed of academics from all knowledge areas.

A distinction between programme (programa) and course (curso)

CAPES uses the Portuguese terms programa (programme) and curso (academic course or
programme) in a specific way. A programa comprises the staff, infrastructure and activities
associated with the provision of postgraduate education in a specific field, whether at
master’s level, doctoral level or both. It is the principle unit of analysis for the periodic
evaluations of postgraduate provision. Of the 3 472 stricto sensu academic postgraduate
programas evaluated in the most recent CAPES four-yearly review in 2017, just over 60%
combined both master’s and doctoral provision, 37% involved only Master’s provision and
2% involved only doctoral provision (CAPES, 2017;2}). The term curso is used to refer to
a single course of study at a particular level: a master’s, Professional Master’s or doctorate.
A programa may thus contain two cursos (a master’s and a doctorate) or effectively be
synonymous with a curso, when only one type of curso is provided. For the sake of clarity,
this chapter uses the English terms programme and course, to allow a distinction to be
made where necessary.

The sections that follow review, in turn, the strengths and weaknesses of the CAPES
evaluation processes for approving new courses (Avaliagdo de Propostas de Cursos Novos,
APCN) and periodic review of established programmes (Avaliagdo Quadrienal).

CAPES: approval of new postgraduate courses

Relevance: rationale and objectives of the current system

In Brazil, academic postgraduate education in all types of higher education institution? is
collectively considered as part of a National System of Postgraduate Education (SNPG),
which — as a system - has the aim of training highly quality teaching and academic staff;
training (highly) qualified staff for non-academic sectors and strengthening the country’s
scientific, technological and innovation capacity more generally. CAPES states that the
evaluation system it operates is designed to certify the quality of postgraduate education in
the country, as a basis for allocation of publicly funded scholarships and research funding,

140 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

and to identify regional disparities strategic knowledge areas as a basis for strategic actions
to address such gaps (CAPES, 2018)3)). As such, the CAPES evaluation system as a whole
serves at least three purposes:

1. It is a mechanism for ensuring the quality of postgraduate training (and thus — in
theory - the quality of the human resources trained) as a form of guarantee for
students and their future employers;

2. The results of the evaluation process (in particular the periodic reviews) provide
objective criteria for the allocation of public funding for researcher training
(notably grants to master’s and doctoral students) and research projects (with the
implicit expectation that the high-quality programmes identified will make good
use of public resources);

3. The results of the evaluation also identify how well the country is developing
research capacity in different scientific fields and across the territory of the Union,
allowing corrective policy measures to be developed as necessary.

The specific approval process for new courses (APCN) is designed to ensure only academic
teams with demonstrated expertise, a proven track record of quality research and adequate
facilities are authorised to provide academic postgraduate education. Course proposals are
assessed by a field committee composed of academic peers from the field in which the
course seeks to operate. Following a standard assessment and validation process, new
courses are formally approved if they score at least three on a nominal scale of one to five’,
taking into account a range of variables discussed below. Only once approved can courses
recruit students; obtain national recognition for their diplomas, and obtain funding from
CAPES for student scholarships and institutional capacity building®.

The APCN process consciously sets a comparatively high bar for entry into the system of
academic postgraduate training and for the creation of doctoral training provision in
programmes that already operate a master’s level. In so doing, it seeks to maintain high
minimum standards for postgraduate education, protect students against poor quality
provision and ensure efficient targeting of public funding. During the review visits, the
OECD team noted a high degree of support for the principle of maintaining a high threshold
for entry into the academic postgraduate education system.

Effectiveness: quality indicators used or generated

CAPES evaluations of postgraduate courses and programmes rely to a large extent on
qualitative assessments undertaken through peer review. These qualitative assessments,
which may take into account quantitative data, are ultimately translated into a single score
(conceito) attributed to courses and programmes. Standing evaluation committees
composed of Brazilian academics in specific disciplines are established for 49 scientific
fields. The work of each committee is coordinated by a “field coordinator’, elected by the
academic programmes in the field in question for a renewable term of four years. The field
coordinator and the members of the field committee undertake their work in the CAPES
evaluation processes on a voluntary basis, alongside their main academic jobs.

The field committees are responsible for assessment new courses and undertaking the
periodic reviews (discussed below). Following initial eligibility checks by CAPES staff,
the relevant field committee assesses proposals for new courses using a standard set of
criteria for the field. The topics assessed follow a standard model developed by CAPES,
but with specific assessment and judgement criteria, and the weighting of individual
variables, tailored to each field by the field committee in question. The field committee

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 141

collectively assesses proposals in relation to the assessment topics and assessment criteria
to provide ratings for each topic on a five-point scale from “very good” to “deficient”. The
ratings for the different topics are combined to generate an overall assessment score for the
programme. Programmes that are finally approved are attributed an initial default CAPES
rating (Conceito CAPES) of three out of five (three being the minimum quality threshold
required). Programmes are only attributed higher scores following a periodic review.

As illustrated in Table 6.1, the evaluation criteria for academic courses (academic master’s
and doctorates) for new course proposals include the relevance of the new course to
national and institutional development; the design and proposed scale of the course; the
qualifications and scientific output of the staff involved and their planned involvement in
the course; and available infrastructure. The same criteria are also used to evaluate
proposals for new Professional Master’s courses although with specific evaluation criteria
modified to take account of the more practical orientation of these courses and their closer
links to the world of work. CAPES regulations (CAPES, 2017, p. art.6;4}) specifically
highlight that professional programmes are likely to require a different staff profile. They
recommend giving weight to the professional experience of teaching staff, even if they do
not hold a PhD, and reducing the emphasis on staff being full-time, given that many will
teach alongside other professional roles.

Table 6.1. Criteria for evaluation of new postgraduate course proposals

Criteria
Fit with institutional development Consistency of proposal to the Institutional Development Plan (PDI) of the proposer
plan and commitment of the institution's leaders to the initiative.
Consistency of proposal and Staff qualifications.
programme design Relevance of ongoing research activities by the team proposing the programme.
Appropriateness of curriculum structure to subject of programme.
Clarity of proposal in relation to Selection criteria for students.
students and planned graduate Number of study places (vagas).
profile Fit of graduate profile with national priorities and needs.
Academic capacity Evidence that the team proposing the programme has academic, didactic, technical
and / or scientific competence and qualifications linked to the objective of the
proposal.
Permanent staff Demonstration that an adequate number of permanent staff with exclusive dedication
are allocated to the programme and will be able to deliver the type and volume of
training proposed.
Scientific output of staff Indication of a maximum of five research outputs for each permanent staff member for
the last five years.
Infrastructure Adequacy of the educational and research infrastructure: physical facilities,

laboratories, experimental facilities and library.
Adequacy of computer equipment, network access and multimedia information
sources for teachers and students.

Adequacy of secretarial infrastructure and administrative support.

Source: Adapted from Portaria CAPES n° 161/2017, article 4 (CAPES, 2017,4)).

The field committee assesses these dimensions based on the electronic application from the
proposing institution (submitted through the CAPES Sucupira platform), with the option
to request a site visit if considered necessary. The OECD team understands that site visits
as part of initial course approval, although formally provided for, are comparatively rare in
practice®.

The criteria examined in the process for approval of new courses cover a wide range of the
variables that might reasonably be expected in an ex-ante assessment of a proposed

142 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

postgraduate programme. The criteria focus on most key factors that might be expected to
contribute to the quality of the future training provided. Nevertheless, there is scope to
review — and certainly to substantiate better - the prominence and weight attributed in the
evaluation template to the different factors considered.

The current evaluation system attaches considerable weight to the status and intellectual
outputs of the staff who will be involved in the proposed course. This is entirely consistent
with the objective of checking that adequate conditions are in place to allow students to
have access to knowledgeable teachers and mentors and receive their training in an
environment where high-quality research is undertaken and valued. This reflects a model
of academic postgraduate education that views research culture and peer effects among
individuals involved in research as key contributors to the learning and scientific
development of students. In placing such emphasis on these factors, however, there is a risk
that other variables affecting the quality of the training offered are attributed too little
attention, particularly at master’s level and in professionally oriented programmes.

The key concerns of the OECD team relate to the comparatively limited attention attributed
to, first, the relevance of new courses to national or regional needs and developing
knowledge areas and, second, the design of the training programme, and support and
personal development opportunities offered to students.

Under the section dealing with students, the existing standard evaluation template for new
courses includes an assessment of relevance of the “graduate profile” the proposed course
is intended to generate — in other words, the types of knowledge and skills graduates are
expected to possess (this is also a consideration in the evaluation instruments used in
SINAES (INEP, 2017,5]). However, this fundamental issue is considered under the same
broad heading as practical issues like student selection and the number of study places.
Although the coherence of the proposed course with the Institutional Development Plan
(PDD of the host institution is assessed, there is no explicit assessment of the relevance of
the course to the needs of Brazil, in terms of knowledge development and highly qualified
human resources. There is no obvious place in the current framework where the
contribution of new courses to new or emerging fields of knowledge is assessed.

These problems are compounded by the fact that assessments are carried out exclusively
by academic staff from a specific discipline, using largely traditional measures of academic
performance. While academics in a given field may be expected to have a good
understanding of the developments in that field in an international context, particularly in
less applied areas, they may, understandably, have less understanding of how knowledge
and skills in the field can contribute to national development goals or respond to societal
challenges. There is scope to include more perspectives from non-academic bodies in this
aspect of the assessment.

Similarly, although five of the 49 academic fields are nominally classified as inter-
disciplinary (biotechnology, environmental science, education, material science and “inter-
disciplinary’’), the strong focus on traditional disciplines and scientific output in these
disciplines may create barriers to new courses in innovative fields of study that may
ultimately be important for the future of Brazil’s postgraduate training system. The risks of
working in disciplinary silos are by no means unique to Brazil, but do warrant further
attention in the way CAPES evaluations are structured and organised.

The second key issue that deserves greater attention in the assessment of new programmes
is organisation of training and support for students. At present, the CAPES evaluation
template includes an assessment of the “appropriateness of curriculum structure to subject

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 143

of programme”’, but little obvious room to assess how the training programme will help to
develop students’ knowledge and skills and monitor their progress. Across the OECD,
higher education institutions and research funding bodies have increasingly focused on
developing postgraduate training with a greater explicit focus on helping students to acquire
research skills and transversal competencies (in collaborative working, communication,
project management, entrepreneurship, for example) that they can exploit subsequently in
a wide range of settings. Evaluation systems in other higher education systems do place
more emphasis of these issues’.

In the discussion of quality indicators, it is important to acknowledge that the current
CAPES assessment system has developed distinct criteria to be applied in the evaluation of
Professional Master’s courses. In particular, the criteria for this type of course take account
of the different staff profile required to successfully implement more applied forms of
training. Between 2010 and 2017, the number of Professional Master’s courses in Brazil
increased from 247 to 703, suggesting that the authorisation system is functioning for this
type of provision. However, developing appropriate quality criteria for applied research
and postgraduate programmes has proved challenging in all OECD higher education
systems and there is certainly scope for ongoing mutual learning. Within Brazil, it is
important to monitor the implementation of existing Professional Master’s programmes, to
discuss strengths and challenges with programmes, students and industry and public sector
partners and to ensure lessons learnt feed back into the evaluation indicators used.

A final consideration about the indicators used in assessment of proposals for new courses
is the absence of an explicit requirement for a course or programme development plan with
measurable, time-bound targets. Requiring programmes to develop such a plan and
establish clear targets would create a useful reference for subsequent periodic reviews.

Effectiveness - division of responsibilities

A defining characteristic of the CAPES evaluation system is the strong role of academic
peers in both defining the evaluation criteria and undertaking programme evaluations. Field
coordinators and committees have leeway to adapt commonly agreed evaluation templates
to the requirements of their specific fields, by defining field-specific assessment criteria
and adapting weightings between broad evaluation criteria. In practice, field committees
stick very closely to the standard evaluation template, but adjust specific evaluation criteria
for individual topics. The strong involvement of the academic community in both policy-
setting and implementation, as well as the flexibility afforded to field committees in the
process, have contributed to the widespread acceptance of the CAPES evaluation system
and a shared sense of “ownership”. This contrasts with the evaluation processes for
undergraduate programmes implemented as part of SINAES, which are widely perceived
as top-down.

Despite the strengths of the current division of responsibilities within the CAPES
evaluation system, the operation of an evaluation system that relies heavily on the voluntary
contribution of academic staff organised in discipline-specific field committees is not
without problems.

First, there is the practical issue of the availability and commitment of academic peers.
Although academics involved in the CAPES evaluation process consulted by the OECD
review team felt the time and effort required of them for the current system for approval of
new programmes remained reasonable, they highlighted that the CAPES system as a whole
is becoming unmanageable for field committees, as the number of postgraduate

144 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

programmes increases. We return to this issue in the discussion of the four-yearly reviews
below.

Another potential risk with the current system is that authorisation to start a new academic
postgraduate programme depends to large extent on the opinion of academics who work in
“rival” postgraduate programmes in the same field and who may have an interest in
restricting expansion of provision to limit competition for students and research funds. In
practice, the OECD review team found no evidence that this potential conflict of interests
has led to any undue restrictions on the creation of new programmes. The number of
postgraduate programmes has increased considerably over the last decade. Moreover,
evaluation criteria for new proposals are clear, field committees need to justify their
evaluation scores in detail, the final evaluation score is validated by CAPES’ inter-
disciplinary Technical and Scientific Council, and transparent procedures exist for
proposing institutions to appeal against decisions.

More seriously, as highlighted above, the reliance on disciplinary committees composed
exclusively of Brazilian academics risks creating an excessively narrow academic focus in
evaluations. While scientific excellence and traditional measures of academic output
remain the basis for postgraduate education, it is important to complement assessment of
this basis with perspectives from outside academia, to ensure the development of
postgraduate education responds to broader national and regional needs. Equally, as
highlighted above, it is crucial that there is room for innovation in the definition of study
fields and the way programmes are implemented.

On a practical level, the current process for the evaluation of new courses involves limited
or no direct interaction between those proposing the new courses and those evaluating the
proposals. This may be justified by the limited availability of time and resources and a
desire to ensure the evaluation is independent and transparent. Nevertheless, the CAPES
evaluation system is notable for being largely “paper-based”. Other quality assurance
systems tend to employ site visits, or at least, as in the Programa Nacional de Posgrados
de Calidad in Mexico, an interview with the course coordinator as part of the initial
authorisation or accreditation process (CONACYyT, 2015,6)).

Effectiveness: use and effects

As noted above, a successful CAPES evaluation is a prerequisite for all new academic
postgraduate courses to begin operation. On passing the initial evaluation process, new
courses are attributed a provisional evaluation rating of three out of five. On this basis, they
have access to CAPES funding for capacity building and student scholarships. Funding for
grants is allocated by CAPES to the programme, which is then responsible for awarding
scholarships to students.

The results of the entry evaluation for new courses are made public on the CAPES website
and are used by courses in their marketing and student recruitment processes. For
understandable reasons, the approval of new courses, which are evaluated in varying
numbers every year, does not attract as much public attention as the results of the four-
yearly periodic evaluations discussed below and which cover the entire stock of
postgraduate programmes.

Efficiency and cost-effectiveness

Academics involved in the CAPES field committees consulted by the OECD review team
tended to indicate that the time and financial resources invested in the evaluation of new

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 145

courses were proportionate to the goals of the system and remained manageable in light of
the average number of new proposals received annually. Although the costs associated with
initial approval of postgraduate courses have not been made available to the OECD at the
time of writing, the absence of systematic review visits and the use of academic field
committees who work on a voluntary basis clearly limit costs for the Brazilian State.

The Review team understands that no assessment of the value of the time dedicated to
evaluation of new courses by academic staff in the field committees — and thus also the cost
to their home institutions - is currently available. Given the comparatively rapid rate of
expansion of postgraduate provision in Brazil in recent years and the related increase in the
number of proposals for new courses, it will be important to develop a better understanding
of the number of person-hours used in the evaluation process and the associated costs.

Given the concern in Brazil to maintain a high quality-threshold for entry of new courses
to the academic postgraduate system in the country, the existing system of systematic peer
review for all new programmes appears to be appropriate in the current Brazilian context.
In the longer term, as the scale of the postgraduate system continues to evolve, it may be
desirable (or necessary) to move away from programme-level initial accreditation to allow
institutions that meet specific conditions and have adequate institutional quality assurance
processes to launch academic postgraduate programmes under their own authority. Such
models of institutional self-accreditation exist in many mature higher education systems,
although the OECD team recognise that such an approach may not yet be appropriate for
an expanding system such as that in Brazil.

CAPES: four-yearly programme reviews

Relevance: rationale and objectives of the current system

Every four years®*, CAPES implements a comprehensive evaluation of all academic
postgraduate programmes that have already been accredited and been in operation
sufficiently long for students to have produced academic results. Although the specific
objectives of this process are not formulated very explicitly in the relevant secondary
legislation, the four-yearly reviews appear to fulfil a double role:

e They provide a means to ensure postgraduate programmes (continue to) meet at
least minimum defined quality standards, as programmes scoring less than three
out of five lose CAPES funding and the national validity of their diplomas, and;

e The reviews provide an incentive for programmes to strive for improvement — as
measured by the defined criteria — as programmes can obtain a higher score (than
that awarded in the initial approval process or in the previous round of periodic
reviews) and thus greater prestige and, potentially, greater funding.

From a quality assurance perspective, the reviews are in practice very much focused on
external assessment and ensuring accountability, with limited or no focus on supporting
programmes to improve (quality enhancement).

Effectiveness: quality indicators used and generated

As for the evaluation of proposals for new courses, the four-yearly reviews are coordinated
by CAPES, but undertaken by the 49 field committees under the leadership of their field
coordinator. The field committees draw on information on staff, students, graduates and
details of scientific outputs reported by each postgraduate programme through the online
Sucupira platform as a basis for their assessment of each programme. As in the case of the

146 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

assessment of new courses, field committees use a standard evaluation grid which they
adapt to the specificities of their field, in particular through formulating specific evaluation
criteria for each topic and adjusting the weights between topics.

As seen in Table 6.2, the assessment includes a review of the programme proposal and its
relevance, although this is not attributed any points in the final score. The criteria relating
to staff are similar to those used in the evaluation of new courses, but verified using data
from Sucupira. Similarly, most of the criteria relating to students are based on quantitative
data reported by the programmes.

The quality of student publications (including published dissertations and theses) and the
quality of the academic output of staff in academic journals are assessed using a standard
classification of publication “vehicles” (from international peer-reviewed journals to
university online publications), recorded in an online database called Qualis. Part of the
work of each field committee each year is to review an established classification of
publication vehicles relevant for their field and attribute a “quality rating” on a seven-point
scale (Al, A2, B1, B2, B3, B4, B5), where Al typically includes the most prestigious
international journals in the field with high impact factors. Citation impact, assessed
through mechanisms such as the Scopus database and citation scores such as the h-index’,
is a significant criterion in the rating of journals in Qualis in many CAPES fields. However,
while the use of impact factors is well-established, but not uncontested, in the hard sciences,
there is an ongoing debate in Brazil, as in other countries, about the extent to which such
measures capture the impact and relevance of work in the social sciences, humanities and
arts.

The development of the Qualis classification database means that publications produced
by each programme (and reported in Sucupira) are automatically attributed a quality rating
on the basis of the assigned rating of the publication vehicle used. As the Qualis
classification is undertaken for each field, the same journal may have a different rating in
different fields. Individual fields have also developed Qualis-like rating systems for artistic
and technical outputs, although these systems are less well established and more complex
to implement.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 147

Table 6.2. Periodic programme evaluation: criteria for academic programmes

Criteria Weighting
(range)
Programme 1.1. Coherence, consistency, comprehensiveness and “currentness” (atualizacao) of 0
proposal the priority research fields, lines of research, projects in progress and curricular
proposal.

1.2. Future planning for the programme taking into account challenges for the
knowledge field in terms of knowledge production, training, social engagement and
the destinations of graduates.

1.3. Infrastructure for teaching, research and outreach / engagement.

Academic staff 2.1. Profile of the academic staff, considering levels of qualification, diversification in 10-20 %
the origin of training, ongoing training and experience and the compatibility of these
with the programme proposal.

2.2. Adequacy and time commitment of permanent teachers to research activities and
the training programme.

2.3. Distribution of research and training activities among the staff involved in the
programme.

2.4. Contribution of programme staff to undergraduate teaching and / or research
activities, paying attention to the repercussion that this item may have on the training
of future participants in the postgraduate programme (only when there is a direct link

with an undergraduate programme).

Students, theses 3.1. Number of theses and dissertations defended in the evaluation period, in relation 30-35%
and dissertations to the number of permanent teaching staff and the size of the student body.

3.2. Distribution of the focus of theses and dissertations defended in relation to the
profile of teaching staff.

3.3. Quality of theses and dissertations and contribution of the academic output of
undergraduate (if the HEI has undergraduate courses in the area) and postgraduate
students to the overall output of the programme, as measured by publications and
other indicators relevant to the field.

3.4. Efficiency of the programme in the training students: time taken for graduation of
master’s students and doctoral candidates.

Scientific outputs 4.1. Quality rated publications by permanent staff member. 35-40%

4.2. Distribution of quality rated publications in relation to the permanent teaching staff
of the programme.

4.3. Technical output, patents and other outputs considered relevant.
4.4. Artistic outputs, in areas where such output is relevant.
Social engagement 5.1. Insertion and regional and (or) national impact of the programme. 10-20%

and impact 5.2. Integration and cooperation with other research and development programmes
and professional development related to the area of knowledge of the programme,
with a view to the development of research and postgraduate studies.

5.3. Visibility or transparency given by the programme to its performance.

Source: CAPES (2017) Regulamento para a Avaliagao Quadrienal 2017 (2013-2016) Programas académicos
e profissionais. (CAPES, 2017(7))

In contrast, no standardised classification system exists for books or book chapters
published by academic staff (or students) in programmes. This means, for fields where
books are a major vehicle for intellectual output, field committees have to assess books and
book chapters individually. Typically, programmes are invited to identify books and book
chapters that they believe meet particular quality criteria established by the field committee
and these are then all reviewed quickly. Books and book chapters identified as having
particular merit are then read in full by members of the field committee. The OECD team
understands that the assessment of books and book chapters represents one of the largest
calls of the time of members of some field committees (notably in the humanities, social
sciences and some of the hard sciences).

148 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

The social engagement and impact (inser¢do social) of programmes is reviewed in a
qualitative fashion on the basis of documentary evidence submitted by programmes. Some
of the field committees examine the destination of graduates under the topic of “Insertion
and regional and (or) national impact of the programme”. However, it is not clear how this
assessment is made and whether it is based on systematic surveys of graduate destinations.

Considered in the round, the set of indicators used in the CAPES four-yearly evaluations
covers many of the key variables that would widely be assumed to contribute to high-
quality postgraduate provision. It is positive that the evaluation grid, under different
headings, takes into account factors such as staff-to-student ratios, time to graduation and
cooperation networks with external research and non-academic organisations, for example.

However, the most striking feature of the four-yearly reviews is the strong focus on the
scientific output of the academic staff involved in the programmes being evaluated. As
noted earlier in this chapter, the presence of competent researchers is crucial to the capacity
of programmes to share subject knowledge and research expertise with students and create
an environment conducive to the students undertaking their own high-quality research.
However, the CAPES evaluation is — nominally at least — an evaluation of postgraduate
training programmes, not a research performance evaluation like the Research Excellence
Framework (REF) used in the United Kingdom. As such, it is questionable why the system
does not allocate less weight and fewer resources to assessing the performance of staff and
more to assessing the performance of students and outcomes of graduates.

The current system does attempt to measure the quality of dissertations and theses and other
papers and outputs of students. However, this assessment is in most cases based on proxies
provided by a notional quality rating attached to the “vehicles” in which the dissertations
or theses (or derivatives thereof) are published. It is questionable whether it is reasonable
to expect master’s students or even doctoral candidates to be publishing outputs in journals
at a similar level to established academic staff. While postgraduate students do publish in
high-quality academic publications, they are typically in a minority in most established
higher education systems. Other forms of publication — such as non-peer-reviewed online
journals - do not necessarily provide a reliable guarantee of quality.

There is no easy solution to these problems in a system like the current CAPES four-yearly
reviews. Quality assurance systems which rely on on-site review visits (principally at the
master’s level), do sometimes involve a qualitative review of a sample of student
dissertations. Other systems rely on other mechanisms to ensure the quality of postgraduate
student outputs — essentially placing trust in standard processes. Several English-speaking
countries rely heavily on external marking (by academic peers) of papers and dissertations
at master’s level, as a means to assure quality across the system. Many systems — including
Brazil — insist doctoral theses are peer reviewed and finally approved by defence panels
composed of leading academics in the field.

As noted, there is also some attempt in the current CAPES system to assess the destinations
of graduates from programmes. However, on the basis of available evidence, this aspect of
performance is not currently addressed adequately. The ability of graduates from
programmes to find relevant employment in, or outside, the academic sector and draw on
their skills must be assumed — in part at least — to reflect the quality of the training they
have received. It would be desirable to further develop systems in Brazil to allow graduates
to be tracked and to include graduate outcomes more prominently in the postgraduate
training evaluation system.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 149

A final issue that deserves attention in this discussion of indicators is the way in which field
committees identify and assess programmes deemed to be of international quality or
excellence, with strong internationalisation and international engagement (inser¢do
internacional). These are attributed CAPES scores of 6 or 7 and subsequently have access
to additional resources. Each field committee is responsible for establishing transparent
criteria for allocating these top scores. In all cases, programmes are initially scored on a
scale of one to five, and then programmes with doctoral provision that score five (that score
“good” or “very good” on all other criteria) are assessed against additional criteria
understood to indicate international excellence. Common criteria include the amount of
external research funding attracted by the programme, the number and intensity of
international cooperation and the proportion of outputs published in international journals.

Given the assumed link between internationalisation and academic excellence, the principle
of making achievement of the highest scores for academic postgraduate programmes
dependent on objective measures of international activity appears sound. Although the
rigour and appropriateness of the indicators used to measure internationalisation may vary
between fields, the types of measures used appear generally to be appropriate, objectively
measurable and comparable to indicators of internationalisation used in other higher
education systems. In the 2017 four-yearly CAPES evaluation, 184 programmes (5.3% of
academic programmes) achieved a score of seven and 298 (8.6%) programmes achieved a
score of six. It is credible that a higher education system of the size and maturity of Brazil’s
would have such numbers of programmes that could be considered of high quality in an
international context. As discussed below, there is scope to bring more international
perspectives into the assessment of quality and the determination of whether programmes
genuinely deliver international standards of excellence.

Effectiveness: division of responsibilities

As discussed earlier in this chapter, the reliance of CAPES on peer review is both a strength,
for the acceptance and credibility of the system in the academic community, and a potential
risk factor, as the scale of the postgraduate training system in Brazil expands and increases
the burden of undertaking peer evaluations. CAPES has hitherto been successful in
attracting and obtaining the commitment of well-regarded Brazilian academics to work as
part of its field committees, including for the four-yearly reviews. However, some of the
academics involved in CAPES evaluations interviewed by the OECD review team
expressed concern that it was becoming harder to engage academics and — from a purely
logistical perspective — the system of peer review as currently configured is no longer
sustainable.

A second key issue in the current staffing of CAPES evaluation processes is the risk of
endogamy (inbreeding). Even in a country size of Brazil — particularly given the relatively
small size of its postgraduate training system — the number of established academics in a
given field of study is limited. The number working in very high-quality departments and
programmes at an international level is even smaller. As such, there is the risk that the
people making judgements on whether or not a given programme is of international
standard have close connections with the programmes they are judging. It is likely that their
appreciation of the relative merits or deficiencies of the programme is conditioned by the
tight-knit academic community of which they are a part. Moreover, the comparatively small
pool of evaluators and their background may lead the evaluation process to reward
programmes that reproduce existing models of education, rather than innovate.

150 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

It would be beneficial to bring the perspective of international peers into the CAPES
evaluation process, particularly for assessment of those programmes judged to be of
international standing. A large-scale involvement of international peers in assessment
would almost certainly be impractical because of the costs involved, the difficulty of
securing participation and language issues. Nevertheless, some targeted involvement of
academic peers, including through electronic communication, from outside the country
may be feasible.

Effectiveness: use and effects

The results of the four-yearly CAPES evaluations have far-reaching effects. For
programmes that fail to meet the minimum quality standard of three out of five, the
evaluation essentially leads to the closure of the programme. Programmes failing to achieve
a score of three lose their right to CAPES funding and see the national validity of their
diplomas withdrawn. It is understood that students in programmes that achieve scores
below three typically have to transfer to other programmes to complete their studies.

The national research funding council, the Conselho Nacional de Desenvolvimento
Cientifico e Tecnologico (CNPq), also takes into account the association of researchers with
particular evaluated programmes in in the assessment of application for individual research
grants, including the “Research Productivity Grants” (Bolsas de Produtividade em
Pesquisa).

Efficiency and cost-effectiveness

The costs of the current CAPES evaluation system for CAPES itself are comparatively
modest owing to the reliance on the voluntary participation of academics in the field
committees. However, as the system evolves, it will inevitably have to find ways to
maintain the value of peer involvement in the evaluation, while reducing the sheer volume
of work required to evaluate each programme.

The use of Qualis appears to be an efficient and relatively effective way of providing
information on the broad quality of a proportion of the scientific output of programmes. In
contrast, the time and effort dedicated to the review of books and book chapters by some
of the field committees seems disproportionate to the information about the quality of
programmes that is obtained from the exercise.

In parallel, however, the CAPES evaluation system is notable for the absence of site visits
to programmes or interviews with programme coordinators and its almost exclusive
reliance on domestic peers for its review work. These features of the system keep costs
down. Any future decision to increase the use of visits and interviews, or the involvement
of international peers are likely to add to the costs of the system.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 151

6.3. Key recommendations concerning CAPES evaluations

1, Adjust the weighting of evaluation criteria in assessment of new courses to
focus more on relevance, training and continuous improvement

The approval of new postgraduate courses through the systems of peer review currently in
place creates an effective mechanism for assuring the quality of new academic postgraduate
education in Brazil. Nevertheless, the OECD review team considers that the current
evaluation process for new courses could be improved by adopting the following
modifications:

e Revise the structure of the evaluation fiche for new courses to create a more
transparent structure that follows the intervention logic for postgraduate training
programmes, moving from inputs (including institutional context, supervisory staff,
facilities) to processes (programme structure, approaches to incorporating practical
experience, methods for supervision, mentoring and assessment) and expected
outputs (graduation times and rates, graduate profiles) that provides a clearly
formulated and valid rationale for each indicator used.

e Include a separate section in the evaluation fiche on the relevance of the programme
to national development needs, taking into consideration the development of new
scientific areas and the knowledge and skills required for the further development
of the country, including in natural sciences, social sciences and the arts.

e Include a more explicit requirement for a programme development plan for all new
programmes approved, setting out specific and measurable goals over time. This
would act as a reference for subsequent periodic reviews and introduce a clearer
focus on continuous improvement. The approach used by CONACYyT in Mexico
for assessment of programmes for the Programa Nacional de Posgrados de
Calidad (PNPC) might provide some inspiration in this regard (CONACyT,
2015,6}).

2. Bring additional perspectives into the evaluation of new programmes

As argued in the preceding analysis, the current field committees undertaking the
assessment of new programme proposals are composed exclusively of academic peers from
the field in question. To bring a broader range of perspectives to the process and potentially
promote innovation and inter-disciplinary cooperation, CAPES should involve one or more
academics from other academic fields in the field committees undertaking the assessment
of new courses.

In addition, to bring in expertise and perspectives from outside the academic community,
CAPES should consider appointing specialists in economic development and the evolution
of skills and knowledge requirements, as well as representatives of the private economy
and the wider public sector to the Scientific and Technical Council (CTC-ES). If
implemented effectively, this could ensure that final decisions on programme approval take
into account broader national needs and developments.

152 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

3. Maintain programme-level accreditation in the medium-term, but consider
the long-term desirability of transitioning to institutional self-accreditation for
established institutions and programmes

Brazil’s postgraduate education system has grown rapidly in recent years and might still be
considered to be in a phase of consolidation, when compared to postgraduate education
systems in many other OECD and partner countries. In the medium term, it therefore makes
sense to maintain course-level accreditation, to maintain oversight of the continued
development of the system and ensure the promotion of quality. In the longer term, it could
be possible to move to a system of institutional self-accreditation linked to strengthened
model of institutional accreditation (see chapter 7). This would allow universities to start
academic postgraduate programmes if they met certain criteria in terms of staff and profile
and had been judged to have strong institutional quality systems in an institutional quality
review. The provision of publicly funded scholarships and additional programme funding
should certainly remain dependent on positive external evaluation of the programme, in
line with practice in many OECD systems.

4. Clarify the objectives of periodic evaluations and rebalance the focus of
evaluation criteria to include greater focus on student outputs and outcomes

The periodic (four-yearly) evaluations of postgraduate programmes currently devote
disproportionate attention and resources to assessing the outputs of academic staff.
Although the quality of staff is an important factor in the quality of postgraduate
programmes, the CAPES evaluations should focus on assessing the conditions for, and
performance of, postgraduate training, not the research output of academic departments.
The OECD review team therefore recommends:

e Rebalancing the weighting in the evaluation criteria for four-yearly assessments,
by increasing the weight attributed to educational processes, student outputs and
employment outcomes, and reducing the weight attributed to staff outputs.

e Reducing the time and resources allocated to assessment of staff output and
assessing only a limited sample of research output. The Qualis for journal rankings
could be maintained, but should also be reviewed, to introduce more uniformity in
the classification of journals between knowledge fields. Less time should be
devoted to assessment of individual outputs (particularly books and book chapters).
This would contribute to reducing the workload for field committees and making
the entire peer-review system more manageable in the medium-term.

e Systematically including interviews with course and programme coordinators as
part of the periodic assessment of courses and programmes, to gain additional
insights into the operation and performance of the programme and answer questions
arising from documentary evidence.

If a more detailed research assessment exercise is considered necessary to promote quality
in the research function of higher education in Brazil, the relevant authorities should
establish this as a separate, but related exercise, with clear and distinct objectives. All
activities undertaken as part of the CAPES evaluation processes should focus on ensuring
quality and promoting quality enhancement in the postgraduate training system.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 153

5. Ensure those judging whether programmes are of international standing
really have an international perspective.

Given Brazil’s aspiration to develop a world-class postgraduate training system, it would
be valuable to gain an international perspective on the programmes judged nationally to be
among the best in the country. The OECD review team therefore recommends that CAPES
systematically involve non-Brazilian academics in the assessment of programmes pre-
selected by field committees as candidates for being programmes of international quality
or excellence. In light of the number of programmes involved, it is likely to be most feasible
to concentrate this international involvement on programmes proposed for the top score of
seven. It may be possible to organise international peer-review committees who are able to
review synthesised information about the programmes under review in English or Spanish
and potentially conduct group interviews remotely or in person with programme
coordinators.

6. Undertake evaluations of specific components of the CAPES system and
aspects of academic postgraduate provision as inputs to future policy

e Second, there is a wider question relating to the future of academic (stricto sensu)
master’s programmes. As noted, master’s programmes in most OECD countries are
now viewed as either purely professional qualifications (as in the United States), or
an extension and deepening of undergraduate studies, which prepares students for
work in knowledge-intensive sectors (as in most of Europe). A doctorate is regarded
as a prerequisite for an academic or research career in most of the world, including,
increasingly, in Brazil. This leaves the question as to what academic master’s
programmes are for. Is the intention that master’s graduates should go on to
undertake a PhD and work in academia, or should they be prepared for work in the
wider economy? If the latter is the case, it is questionable whether master’s
programmes should continue to be part of the highly academic and research-
focused CAPES evaluation processes (notwithstanding the recommendations about
rebalancing above).

It would be valuable to undertake a systematic evaluation of the role of master’s
education in Brazil, including a specific focus on the profile and effectiveness of
the Professional Master’s programmes created in recent years. This evaluation
should consider, in particular, the destinations of previous graduates from these
programmes and the views of the academic community and private and public
sector employers on the relevance and future role for master’s-level education in
Brazil.

154 | CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION

Notes

' The capacity of higher education institutions to provide /ato sensu postgraduate programmes is
verified through the institutional accreditation and re-accreditation procedures implemented by
INEP as part of SINAES. The operation of Jato sensu “specialisation” programmes more broadly is
governed by a 2007 Resolution of the National Education Council. (CNE, 200794).

> Of 32 European higher education systems examined by the European University Association’s
“Autonomy Scorecard”, only ten require universities to seek prior accreditation to start a doctoral
programme.

3 Including Federal, State and municipal public institutions and private institutions.

4 The OECD understands that the maximum score initially attributed to a new course is three.
Programmes are ultimately rated on a scale of 1-7, where scores 6 and 7 are reserved for programmes
with doctoral provision and that are assessed to be operating at an internationally comparable level
of excellence. Scores 4 and 7 can only be attributed following a full periodic review, once the
programme is well established.

> Several funding programmes are run by CAPES for postgraduate programmes with a CAPES
evaluation score of at least three. For public institutions, the Programa de Demanda Social (DS)
provides funding for student grants and the Programa de Apoio a Pos-Graduagado (PROAP)
provides funding for the programme itself (facilities, project etc.). For private institutions, the
Programa de Suporte a Pos-Graduagao de Instituigées de Ensino Particulares (PROSUP) provides
funding for student grants. Programmes in the public and private sectors that achieve a score of 6-7
(which necessarily have a doctoral programme) can obtain further grant funding from the Programa
de Exceléncia Académica (PROEX).

° The relevant guidelines for approval of new programmes for different areas always suggest that a
site visit may be conducted, but not that visits are systematically a part of initial course approval.
(CAPES, 2018)93)).

7 The Mexican Programa Nacional de Posgrados de Calidad (PNPC), for example, includes a
specific criterion on “follow-up and academic development of students” (CONACYyT, 2015;99)).

8 Until 2013, the periodic evaluations were conducted every three years.

° The h-index is an author-level metric that attempts to measure both the productivity and citation
impact of the publications of a scientist or scholar. The index is based on the set of the scientist's
most cited papers and the number of citations that they have received in other publications. The
index can also be applied to the productivity and impact of a scholarly journal as well as a group of
scientists, such as a department or university or country.

CHAPTER 6. ASSURING THE QUALITY OF POSTGRADUATE EDUCATION | 155

References

CAPES (2018), Sobre a Avaliagao - Objetivos da Avaliagao, CAPES website, [3]
http://www.capes. gov.br/avaliacao/sobre-a-avaliacao (accessed on 17 November 2018).

CAPES (2017), CAPES divulga resultado final da Avaliagdo Quadrienal 2017, CAPES Website, [2]
http://www.capes. gov.br/sala-de-imprensa/noticias/869 | -capes-divulga-resultado-final-da-
avaliacao-quadrienal-2017 (accessed on 17 November 2018).

CAPES (2017), Portaria N° 161, de 22 de agosto de 2017 - Avaliagado de Propostas de Cursos [4]
Novos, APCN, de pos-graduacao stricto sensu. (Ordinance 161 of 22 August 2017 -
Evaluation of Proposals for New Programmes (APCN), postgraduate stricto sensu),
https://capes. gov.br/images/stories/download/legislacao/30082017-Portaria-N-161-de-22-de-
agosto-de-2017.pdf (accessed on 17 November 2018).

CAPES (2017), Regulamento para a Avaliagao Quadrienal 2017 (2013-2016) Programas [7]
académicos e profissionais., Coordena¢gao de Aperfeigoamento de Pessoal de Nivel Superior,
Brasilia, http://capes.gov.br/avaliacao/sobre-a-avaliacao/legislacao-especifica (accessed on
17 November 2018).

CONACYT (2015), Programa Nacional de Posgrados de Calidad (PNPC) Consejo Nacional de [6]
Ciencia y Tecnologia - Marco De Referencia, Consejo Nacional de Ciencia y Tecnologia,
Mexico City.

European University Association (2018), University Autonomy in Europe, (1)
https://www.university-autonomy.eu/ (accessed on 15 October 2018).

INEP (2017), Jnstrumento de Avaliagdo de cursos de graduacao Presencial e a distancia - [5]
Reconhecimento e Renovagao de Reconhecimento (Evaluation instrument for undergraduate
programmes - classroom-based and distance - recognition and renewal of recognition),

Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira, Brasilia,
http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 157

7. Assuring the quality of higher education institutions

Like the external quality assurance systems in many other OECD and partner countries,
the National System for Evaluation of Higher Education (SINAES) in Brazil evaluates both
higher education institutions and individual study programmes within those institutions.
Private and public institutions are subject to periodic re-accreditation, based on on-site
reviews coordinated by INEP. Whereas for private institutions, re-accreditation is a
prerequisite for their continued operation, for legally protected public institutions, the
process is essentially a formality. In both cases, the period for which re-accreditation is
granted varies depending on the status and institutional quality score awarded to the
institution. Institutions are also subject to annual monitoring, based on the average
performance of their programmes in relation to SINAES programme-level indicators and
the results of CAPES evaluations for “stricto sensu” postgraduate programmes. This
chapter examines these processes and provides recommendations for the future
development of the systems in place.

158 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

7.1. Focus of this chapter

The external quality assurance system in Brazil evaluates institutions as well as
programmes

The period for which (re-)accreditation is valid varies depending on the status and
institutional quality score (CI) already awarded to the institution. Universities and
university centres are only re-accredited every eight to ten years, while colleges must be
re-accredited at least every five years.

In addition, institutions are subject to annual monitoring, based on the average performance
of their programmes in relation to SINAES programme-level indicators and the results of
CAPES evaluations for stricto sensu postgraduate programmes. The weighted averages of
the Preliminary Course Score (CPC), discussed in Section 5.2 and, where applicable, the
scores attributed by CAPES for new and existing postgraduate programmes, discussed in
Chapter 6 are used to produce an overall score for each institution called the “General
Course Index” (indice Geral de Cursos, IGC).

Institutional evaluation, including self-evaluation, has a central place in the
original legislation governing the system

The wording of the legislation establishing the SINAES in 2004 recognises the central role
of institutions in structuring and providing higher education, alongside their research and
engagement functions, and acknowledges the importance of institutional autonomy and
profile in allowing HEIs to fulfil their missions. The first article of the law establishing
SINAES states:

The aim of SINAES is to improve the quality of higher education, to expand its
provision, to increase institutional efficiency and effectiveness [eficacia] and
academic and social impact [efetividade], and especially to promote deepening of
the social commitments and responsibilities of higher education institutions,
through developing their public mission, the promotion of democratic values,
respect for difference and diversity, [and] the affirmation of autonomy and
institutional identity. (Article 1 of Law 10861 of 2004 establishing SINAES
(Presidéncia da Republica, 2004/11) bold added by the OECD Secretariat)

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 159

The legislation states that evaluation of the federal higher education system will involve
institutional evaluation, programme-level evaluation and assessment of the performance of
students through ENADE. It places institutional evaluation first and develops objectives
and evaluation criteria for institutional review in more detail than for programme-level
evaluation and ENADE. It specifies ten main dimensions to be taken into account in
internal and external institutional evaluation processes, including the Institutional
Development Plan (PDJ); institutional policies; social responsibility; management;
infrastructure; student support and financial sustainability. The seventh dimension listed is
“planning and evaluation, especially the processes, results and effectiveness of institutional
self-assessment”.

To undertake this institutional self-evaluation, the 2004 law specifies that all HEIs must
create an Internal Evaluation Commission (Comissdo Propria de Avaliagao, CPA). This
body is tasked both with coordinating all internal evaluation processes inside the
institutions and transmitting institutional and programme-level information to INEP, as
input to external evaluation activities. The CPA must include representatives from all
sections of the academic community (different categories of staff) and social partners
(“organised civil society”), and have formal independence from other management and
collegiate bodies in the institution.

Internationally, HEIs have varying levels of autonomy to take responsibility for
self-evaluation and quality assurance

The distinction between a) internal evaluation processes within HEIs; b) external
programme evaluation and; c) external institutional evaluation, as seen in Brazil, is found
in many higher education systems in OECD and partner countries. However, the extent to
which systems rely on each of these three components varies considerably.

The systems of quality assurance in Ireland, England and Scotland, for example, dispense
almost entirely with external programme evaluation and rely instead on internal quality
assurance systems within institutions (self-evaluation), which are verified through external
institutional reviews (QAA, 20182); QQI, 2018;)3}). Most quality accreditation activities in
the diverse quality assurance landscape in the United States also involve institutional
reviews, which verify internal quality processes (Hegji, 2017(4}). Quality assurance systems
in many other European higher education systems, including the Netherlands, Sweden or
Portugal, all include both programme and institutional review in their external quality
assurance systems, alongside internal quality assurance. In all three of the latter countries
there have been initiatives to move to systems based primarily or exclusively on
institutional review (NVAO, 2016;5); UKA, 2018); A3ES, 2018;7)). In contrast, in Mexico,
where there is no comprehensive and compulsory system of external quality assurance in
higher education, existing external quality assurance mechanisms focus primarily on
programme-level accreditation (CIEES, 2018(s)).

Overall, while external quality assurance systems in many countries may initially have
included a strong focus on programme-level review, there has been a general trend among
policy-makers and international bodies working in quality assurance to recommend
increased institutional responsibility for quality and to focus external evaluation efforts
primarily on the institutional level. This is the philosophy reflected, for example, in the
current European Standards and Guidelines for quality assurance in higher education (ESG,
2015;9}), which serve as a reference for quality assurance in the 48 countries of the European
Higher Education Area.

160 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

Despite its legal basis, Brazil’s current quality assurance model gives
comparatively limited responsibility to institutions for assuring their own
quality.

While the letter of the law governing quality assurance in higher education in Brazil accords
a central role to institutional autonomy and self-evaluation, the practical implementation of
the SINAES imposes a complex system external programme-level scrutiny on a three-year
cycle. For institutions that perform poorly in ENADE and on the CPC, this leads to regular
programme-level inspections, using prescriptive processes that limit the room for
manoeuvre for institutions. For institutions that tend to perform well in relation to ENADE
and the CPC, particularly universities and university centres that are only subject to
institutional review every eight to ten years, on-site evaluations by external reviewers are
comparatively infrequent occurrences with limited consequences.

The remainder of this chapter explores these issues, reviewing the processes currently in
place in Brazil to assess the quality of individual higher education institutions, including
ongoing monitoring through the General Course Index (IGC) and periodic re-accreditation
reviews.

7.2. Strengths and weaknesses of the current system
Indicator-based monitoring of institutions: the General Course Index

Relevance: rationale and objectives of the current system

As discussed in Chapter 4, new private higher education institutions, and new campuses of
existing private providers in municipalities outside the location of their headquarters, are
required to obtain accreditation from SERES before they can start operating. Institutions
undergo an on-site review by an external review commission appointed by INEP, which
attributes the institutions and Institutional Score (Conceito Institucional) or Cl, on a five-
point scale. Institutions that receive a score of three or above receive formal accreditation,
and, in the logic of SINAES, this institutional score is the official quality “grade” attributed
to the institution and published on the e-MEC platform.

New public institutions are exempt from this initial accreditation process, as they are
effectively accredited as part of their acts of establishment. Public, like private, institutions
are formally required to undergo renewal of their accreditation through a process involving
another on-site review periodically that leads to a new CI and which we examine below.
However, whereas the re-accreditation process could theoretically lead to the “de-
accreditation” of private HEIs, public institutions are protected by their legal status. This
means re-accreditation is purely an administrative formality for public institutions. Even

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 161

for private institutions in Brazil, the risk of “de-accreditation” appears to be low. Although
clear data on the number of cases of institutional “de-accreditation” have not been made
public by SERES, the OECD review team understands only a handful of private institutions
have last accreditation in the last decade.

All institutions with operational courses are subject to the cycle of programme-level
evaluation through ENADE, on the basis of which INEP calculates the Preliminary Course
Score (CPC), discussed in Chapter 5, by combining ENADE results with other programme
indicators. Once three cohorts of students have graduated (over three years) and, depending
on the subjects in their programme profile, potentially been subject to three cycles of
ENADE, INEP calculates another composite indicator: the General Course Index. This
“Index’’, also on a scale of one to five, is calculated based upon:

e The average CPCs of the last three-year period (in which all subjects have been
subject to a round of ENADE), for the programmes that have been evaluated in the
institution, weighted by the number of enrolments in each of the programmes
included in the calculation;

e The average of the evaluation score of the stricto sensu postgraduate programmes
awarded by CAPES in the last available evaluation round, converted to a
compatible scale and weighted by the number of enrolments in each of the
corresponding postgraduate programmes;

e The averaged (enrolment-weighted) sum of scores from undergraduate and stricto
sensu postgraduate programmes (INEP, 2017;10)).

The General Programmes Index (IGC) provides a single, synthetic, and comparative
indicator of institutional performance. At the time of its creation, the IGC was conceived a
means to allow the Ministry of Education “to identify the most precarious institutions and
focus its attention on them” (Schwartzman, 2013;11)). In this sense, it mirrors the function
of the CPCs in renewal of programme recognition.

Effectiveness: division of responsibilities

The methodology used to calculate the IGC, as well as the calculation and presentation of
results (in e-MEC), are the responsibility of INEP. Institutions bear no responsibility in this
process, apart from participation in the administration of ENADE and in the CAPES
assessments of postgraduate programmes, and in reporting administrative data to INEP —
via the CPA - used in the calculation of the IGC.

Effectiveness: use and effects of the IGC

The IGC score is used by external bodies and the media in reporting about the quality of
higher education in Brazil. It is used in a well-known institutional ranking published
annually by the Folha de Sao Paulo, one of Brazil’s leading newspapers (Folha de S.Paulo,
2018,12}). Notwithstanding its original purpose, the IGC is widely perceived as a visible
public signal of institutional quality that institutions themselves feature in advertising. It
likely also bears upon the equity performance of publicly listed for-profit firms. One private
university states on its website, for example:

The Universidade Positivo (UP) has been rated, for the sixth time running, the best
private university in Parana State, with a score of 4 in the Indice Geral de Cursos
(IGC), which goes from I to 5. Revealed last Monday (27) by the Instituto Nacional
de Estudos e Pesquisas Educacionais Anisio Teixeira (INEP) [and] the Ministry of

162 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

Education (MEC), the IGC is the official indicator of the quality of higher
education institutions in Brazil. (UP, 2018/13)

The real signal value of the IGC as a quality indicator for consumers is, perhaps, quite
limited. While IGC scores range, in principle, from one to five (after rounding), scores of
one are virtually unknown, and nearly all scores cluster at values of three and four. In 2016,
93% of universities and 96% of university centres received scores of three or four (INEP,
2017114)). Setting aside the validity or reliability of the IGC, it is clear that its discriminating
power for non-college institutions is low.

Although the reputational effects of the IGC can be important, it is not an indicator that is
likely to have an impact of how institutions understand and manage the quality of the
education that they provide. The IGC does not introduce new performance information for
institutional leaders. Rather, it is a (weighted) summation of already-existing programme-
level information. As such, it replicates the measurement problems of the CPC.

Although the IGC may have been created, like the CPC, as a means for public authorities
to identify weak institutions, it is not mentioned in the most recent secondary legislation
providing implementation rules for SINAES (Presidéncia da Republica, 2017115); MEC,
2017,16]). Unlike the CPC, it does not currently play a direct role in the process of regulation
of higher education institutions. The relevant legislation states that all institutions must
undergo an on-site visit for institutional re-accreditation.

Institutional reviews: re-accreditation

Relevance: rationale and objectives

Entirely separate from the assessment-based IGC — and potentially at odds with it — are the
on-site institutional reviews that are periodically undertaken for the re-accreditation of
higher education institutions. The same process is also used for changes in the institutional
status - transformagao de organizagdo académica - of colleges to university centres, or
university centres to universities. The process of institutional re-accreditation generates a
new institutional score (Conceito Institucional) or CI, that replaces the CI attributed during
initial accreditation as the most recent quality score for the institution. The CI score is used
by SERES to determine whether the institution is permitted to continue to award recognised
degrees, and when the next cycle of re-accreditation must take place. For university
institutions the duration of re-accreditation may range between five and ten years,
depending upon their CI score. For colleges and university centres, the range is from three
to five years, as shown in Table 7.1.

Table 7.1. Duration of accreditation and re-accreditation

Type of institution Institutional score required Duration of accreditation
Colleges and University Centres Cl3 3 years
cl4 4 years

Cl5 5 years

Universities Cl3 5 years

cl4 8 years

Cl5 10 years

Source: MEC (2017) Regulatory Ordinance No 1, 3 January 2017 establishing duration of the validity of
regulatory acts for accreditation and re-accreditation of HEIs (MEC, 2017117).

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 163

Effectiveness: division of responsibilities

Re-accreditation, like accreditation, is the joint responsibility of higher education
institutions, INEP, and SERES. However, as higher education institutions are fully
operational, the institution’s Internal Evaluation Commission (CPA) plays an active role in
preparing an institutional evaluation report, based on consultations with academic staff and
management. INEP is responsible for organising the on-site review process, and SERES
with taking regulatory action of the basis of evaluation information provided by the review.

Effectiveness: quality indicators used and generated

The Institutional Score (CI) that results from the review process is generated based upon
the evaluation commission’s scores for up to 50 indicators set out in the relevant INEP
evaluation instrument (INEP, 201711s)). The focus of these indicators, structured into the
same five axes as for accreditation, is principally the institutional development plan (30%),
the institution’s infrastructure (30%), and its management policies (20%).

Table 7.2. Number and weight of indicators for institutional re-accreditation and change of
institutional status

Axis Number of indicators Weight
Planning and institutional evaluation i) 10
Institutional development 7 30
Academic policies 12 10
Management policies 8 20
Infrastructure 18 30
TOTAL 50 100

Source: OECD calculations based upon INEP evaluation instruments. (INEP, 2017,18}).

Like the other on-site review processes (such as recognition), the re-accreditation review
process is focused on input and process, rather than outputs or performance, and reviewers
are responsible for scoring qualitative indicators on a five-point scale. The review team’s
categorisation of the indicators in the evaluation instrument is shown in Table 7.3. Given
that the process of re-accreditation necessarily focuses on institutions that are already
operating, with graduating students and graduates, there is scope to include greater
consideration of outputs (graduates and evidence of their learning outcomes) and outcomes
(graduate destinations) in the institutional assessments at this stage.

Table 7.3. Type of indicators used in the re-accreditation process

Total number of indicators 50
Total input 35
Total process 15
Total output 0

Source: OECD calculations based upon INEP evaluation instrument (INEP, 2017r18}).

The current evaluation instrument for institutional re-accreditation devotes comparatively
little attention (in terms the number of indicators and judgement criteria) or weight to
assessment of the internal evaluation capacity of institutions. The first “axis” of the
assessment framework focuses on institutional planning and evaluation activities does
contain five indicators relating to internal evaluation process and the quality of the internal

164 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

evaluation report produced by the CPA. Many of the factors identified in the individual
judgement criteria for these indicators would appear highly relevant for the assessment of
internal quality management practices, although these factors are not developed and
explained in detail. However, as these issues are embedded in few individual indicators in
an axes that contributes only 10% of the overall institutional score, the quality of internal
evaluation capacity does not currently play a major role in whether an institution is re-
accredited or what institutional score they receive.

Effectiveness: use and effects

Owing to the schedules for re-accreditation highlighted above, the CI score is not calculated
and reported on an annual basis, but rather with a periodicity that may range from three to
ten years. In light of its infrequency, and perhaps because it is not linked to student
outcomes as observed in ENADE, the CI score appears to function solely as a regulatory
input, and not as a public signal of institutional quality. In the course of stakeholder
meetings with the OECD review team, the CI score was not identified as a measure to
which institutions managed or adapted their performance. The ongoing monitoring of the
Institutional Development Plan through internal evaluation processes (self-evaluation) was
cited as an important feature of the re-accreditation process.

However, there was variation in the extent to which it was considered to have generated
useful reflection within institutions or contributed to the development of a genuine quality
culture. Some CPAs reported that their HEI found the self-evaluation activities to be a
compliance activity in which few colleagues wished to participate. Other institutions found
the obligation to create development plans and undertake a structured analysis of
institutional performance spurred useful quality discussions that would have otherwise not
have occurred.

7.3. Key recommendations concerning institutional review

In the view of the OECD Review team, the processes of institutional quality assurance that
result from periodic re-accreditation and regular reporting of IGCs need fundamental
improvements. Based on the analysis above, the team makes the following
recommendations:

1. Reduce the period of re-accreditation for universities and university centres

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 165

2. Reduce the weight attached in institutional re-accreditation reviews to input
and process indicators that measure basic supply conditions for higher
education

There is scope to rebalance the weights attributed to the evaluation indicators used at the
stage of institutional re-accreditation, away from inputs and towards processes and outputs.
A first aspect of this is to remove indicators that measure basic supply conditions for higher
education, such as infrastructure and equipment and general management policies. The
availability of suitable infrastructure to supply each undergraduate programme is verified
through the programme-level recognition and renewal of recognition processes, while the
most general institutional policies are unlikely to change — or need to change - considerably
over time. It is therefore wasteful to devote resources to re-evaluating and re-scoring these
kinds of variable through the re-accreditation review. The inclusion of these indicators also
reduces the proportional weight attributed to factors that are important to verify in re-
accreditation, such as educational results and institutional performance.

3. Increase the weight attributed to outputs and outcomes in periodic
institutional assessment

Evidence about educational results and institutional performance is neglected in the current
system of institutional re-accreditation. While processes of accreditation cannot take into
account programmatic and institutional performance, re-accreditation can — but does not.
Institutions should be able to graduate most students who begin their studies and they
should do so in a timely way. Those who graduate should be able to find employment,
preferably in fields related to their area of study — and most certainly so if their studies have
a career orientation — whether accounting, civil engineering, or nursing.

SINAES was based in a view that quality assurance could proceed through coordinated and
complementary processes — including institutional self-assessment, detailed on-site
institutional reviews carried out by peer reviewers, and through the implementation of
learning assessments (ENADE) and use of related indicators. The processes of evaluation
that have evolved are not complementary to one another. On-site reviews and performance
indicators generate information about institutional quality that is either incommensurable
or contradictory.

Quantitative programme and institutional indicators should ideally focus on the outputs and
outcomes of higher education, while on-site reviews conducted by peers would helpfully
focus on the inputs and processes that generate the outputs and outcomes observed in
indicators. For example, indicators focused on outputs or outcomes, such as graduation
rates, would be complemented by an on-site review process that examines the conditions
that affect variation in these rates. Thee conditions include student advice and mentoring
processes; how institutions identify students at risk of falling behind or dropping out; the
social or psychological; and academic support services provided to students at risk.

4, Increase incentives for institutions to take a strategic view of quality

166 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

comprehensive and strategic view of their institution, its profile, and the quality of its
educational programmes. Universities are, notionally, institutions that provide research-led
teaching, and should be able to give an account of where and how undergraduate education
is joined up to their research mission. Institutional Development Plans examined by the
OECD review team, however, do not show evidence of this. It would be valuable to provide
incentives to institutions to develop more meaningful PDIs with a stronger focus on how
quality across a range of dimensions can be maintained and enhanced. One way to do this
is to make assessment of internal quality policies and practice a much bigger part of the re-
accreditation process, through greater weighting in the relevant evaluation instrument for
on-site reviews.

5. Move to a system where institutions that can demonstrate strong internal
quality assurance capacity and a proven record of delivering quality can
accredit (authorise and recognise) their own programmes

CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS | 167

References

A3ES (2018), Acreditagao e Auditoria - A3ES, géncia de Avaliacao e Acreditagao do Ensino
Superior (A3ES), http://www.a3es.pt/en/node/83 (accessed on 18 November 2018).

CIEES (2018), Evaluacion de programas, Comités Interinstitucionales para la Educacion
Superior, https://ciees.edu.mx/ (accessed on 18 November 2018).

ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education
Area (ESG).

Folha de S.Paulo (2018), Ranking Universitario Folha - 2017, Folha de S.Paulo,
http://ruf.folha.uol.com.br/2017/ranking-de-universidades/ranking-por-ensino/ (accessed on
17 November 2018).

Hegji, A. (2017), An Overview of Accreditation of Higher Education in the United States,
Congressional Research Service, Washington, D.C., http://www.crs.gov (accessed on
17 November 2018).

INEP (2017), Inep divulga Conceito Preliminar de Curso e Indice Geral de Curso de 2016
(INEP reveals the Preliminary Course Score and General Course Index for 2016), INEP 24
November 2017, http://portal.inep.gov.br/artigo/-

/asset_publisher/B4AQV9zFY 7Bv/content/inep-divulga-conceito-preliminar-de-curso-e-
indice-geral-de-curso-de-2016/21206 (accessed on 17 November 2018).

INEP (2017), Jnstrumento de Avaliagao Institucional Externa - Presencial e a distancia -
Recredenciamento Transformacao de Organizagao Académica (External institutional
evaluation instrument - Classroom-based and distance - Re-accreditation and change of
academic status), Instituto Nacional de Estudos e Pesquisas Educacionais Anisio Teixeira,
Brasilia,
http://download.inep.gov.br/educacao_superior/avaliacao_institucional/instrumentos/2017/IE
S_recredenciamento.pdf (accessed on 11 November 2018).

INEP (2017), Nota Técnica No. 39/2017/CGCQES/DAES (Methodology for calculating the
Indice Geral de Cursos - IGC), Instituto Nacional de Estudos e Pesquisas Educacionais
Anisio Teixeira, Brasilia,
http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_ tecnica _n39

2017 _cgcges_daes_calculo_igc.pdf (accessed on 18 November 2018).

MEC (2017), Portaria N° 23 de 21 de dezembro de 2017 - sobre o exercicio das fungdes de
regulagdo, supervisdo e avaliagdo de instituigdes de educagdo superior e de cursos
superiores (Ordinance 23 of 21 December 2017 regarding the exercise of the functions of
regulation, supervision and evaluation of HEIs and programmes), Diario Oficial da Uniao -
Imprensa Nacional, http://www.imprensanacional.gov.br/materia/-
/asset_publisher/KujrwOTZC2Mb/content/id/1284670/do1-2017-12-22-portaria-n-23-de-21-
de-dezembro-de-2017-1284666-1284666 (accessed on 18 November 2018).

[7]

[8]

[9]

[12]

[4]

[14]

[18]

[10]

[16]

168 | CHAPTER 7. ASSURING THE QUALITY OF HIGHER EDUCATION INSTITUTIONS

MEC (2017), Portaria Normativa N° 1, de 3 de janeiro de 2017 Estabelece os prazos de
validade para atos regulatorios de credenciamento e recredenciamento das Instituigdes de
Educagdao Superior. (Ordinance No. of 3 January 2017, establishing duration of validity of
regulatory acts for accreditation and re-accreditation of HEIs),
http://www.in.gov.br/autenticidade.html, (accessed on 14 November 2018).

NVAO (2016), Assessment framework for the higher education accreditation system of the
Netherlands, Nederlands-Vlaamse Accreditatieorganisatie (NVAO), The Hague,

https://www.nvao.com/system/files/procedures/Assessment%20Framework%20for%20the%2
OHigher%20Education%20Accreditation”%20System%200f%20the%20Netherlands%202016
_0.pdf (accessed on 17 November 2018).

Presidéncia da Republica (2017), Decreto N° 9.235, de 15 de dezembro de 2017 - Disp6e sobre o
exercicio das funcdes de regulagdao, supervisdo e avaliagao das instituigdes de educagao
superior e dos cursos superiores de graduagdo e de pos-graduagao no sistema federal de
ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of
regulation, supervision and evaluation of higher education institutions and undergraduate
and postgraduate courses in the federal education system),
http://www.planalto.gov.br/ccivil 03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on
10 November 2018).

Presidéncia da Republica (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional
de Avaliagao da Educagao Superior — SINAES e da outras providéncias (Law 10 861
establishing the National System for the Evaluation of Higher Education - SINAES and other
measures), http://www.planalto.gov.br/ccivil 03/ ato2004-2006/2004/lei/110.861.htm
(accessed on 10 November 2018).

QAA (2018), Reviewing Higher Education, The Quality Assurance Agency for Higher
Education, https://www.qaa.ac.uk/reviewing-higher-education (accessed on
18 November 2018).

QQI (2018), /nstitutional Reviews, Quality and Qualifications Ireland,
https://www.qqi.ie/Articles/Pages/Institutional-Reviews07.aspx (accessed on
18 November 2018).

Schwartzman, S. (2013), “Uses and abuses of education assessment in Brazil”, PROSPECTS,
Vol. 43/3, pp. 269-288, http://dx.doi.org/10.1007/s11125-013-9275-9.

UP (2018), UP é considerada melhor universidade privada do Parana pelo sexto ano
consecutivo (UP considered the best private university in Parana for the sixth year running),
Universidade Positivo, https://www.up.edu.br/institucional/up-e-considerada-melhor-
universidade-privada-do-parana-pelo-sexto-ano-consecutivo (accessed on
17 November 2018).

[17]

[5]

[15]

]

[2]

[3]

[11]

[6]

[13]

CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE | 169

8. Governance of external quality assurance

Effective systems of external quality assurance for higher education require effective
governance. International experience points to the necessity of three key characteristics in
the design of institutions for the governance and implementation of quality assurance.
First, quality external assurance bodies need to be independent of government and the
demands of party politics, and of the higher education sector itself. Second, responsible
authorities must have sufficient resources - financial, human, and intellectual - to meet
properly their responsibilities. Third, quality assurance bodies should have carefully
developed procedures for engaging with the broader society that higher education systems
have a responsibility to serve. This chapter briefly examines the governance and
implementation of quality assurance in Brazil in light of these considerations and provides
recommendations for the further development of system governance.

170 | CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE

8.1. Focus of this chapter

Effective systems of external quality assurance for higher education require effective
governance. International experience points to the necessity of three key characteristics in
the design of institutions for the governance and implementation of quality assurance. First,
quality external assurance bodies need to be independent of government and the demands
of party politics, and of the higher education sector itself (ESG, 201513; INQAAHE,
20162). Second, responsible authorities must have sufficient resources - financial, human,
and intellectual - to meet properly their responsibilities. Third, quality assurance bodies
should have carefully developed procedures for engaging with the broader society that
higher education systems have a responsibility to serve (CHEA, 2016)3)). Below, we briefly
examine the governance and implementation of quality assurance in Brazil in light of these
considerations.

8.2. Strengths and weaknesses of the current system

SERES, INEP and CAPES together regulate, evaluate and supervise the federal
higher education system

An elaborate system of quality assurance has evolved in Brazil, in which responsibility for
quality assurance rests principally with the Ministry of Education (MEC) and, in particular,
the Ministry’s regulatory department, the Secretariat for Regulation and Supervision of
Higher Education (SERES), and executive agency for evaluation, INEP. In its evaluation
of academic postgraduate programmes, CAPES, which is also answerable to MEC, focuses
on a specific sub-sector of activity operating within an institutional landscape much of
which is ultimately regulated and evaluated by SERES and INEP’.

INEP is a semi-autonomous federal institute (literally an “autarchy”’, Brazilian
administrative terminology), linked to MEC and bears responsibility for planning,
coordinating, collecting, and analysing evidence upon which the quality assurance of
undergraduate education rests. This includes planning and managing the collection of data
for the census of higher education institutions; designing and supervising, and analysing
the results of the ENADE (and ENEM); planning and managing on-site evaluations; and
developing and managing the indicators that inform and support the regulatory work of
government, such as the CPC and IGC.

The evidence of institutional and programme quality produced by INEP’s evaluation work
provides the evidentiary basis upon which SERES supervises and regulates the federal
system of higher education, both federal public and private institutions. SERES
recommends the accreditation, re-accreditation and termination of accreditation of higher
institutions to the National Council of Education, the deliberations of which are
subsequently sanctioned by the Minister of Education.

CONAES formally supervises the evaluation work of INEP

Under the provisions of SINAES, the National Commission for Evaluation of Higher
Education (CONAES) is responsible for proposing and assessing the procedures for the
evaluation of institutions, courses and students; for approving the list of courses to be
assessed by ENADE, and, more generally, for advising on the articulation federal and state
education systems. CONAES is composed of representatives of INEP, CAPES, SERES;
representatives of student, teacher and higher education administrative personnel

CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE | 171

associations; and five external members of recognised expertise appointed the Minister
of Education.

This system has strengths...

There are important strengths to the governance and implementation of quality assurance
in Brazil. For example,

e INEP is recognised internationally as a leading public agency for educational
assessment. Its wide experience with large-scale assessment and its capacity to
manage data collection systems provides the nation’s higher education quality
assurance system with a high level of competence.

e CONAES has succeeded in attracting experts to its council, and through them has
been able to mobilise higher education research from across the nation to inform
the further development of SINAES.

e The basic legitimacy and integrity of the quality assurance system is widely
accepted across the higher education system, by public and private institutions
alike, and by representatives of academic staff and the administrators and owners
of higher education institutions.

e In the course of its implementation, SINAES has used a range of evaluation
techniques — including self-assessment, peer review, and external review grounded
in student assessment - that has been widely welcomed.

e Some higher education institutions in Brazil now closely monitor the experience of
their students and their readiness to participate in external assessments. Others are
making efforts to use compulsory self-assessment and peer-review processes as
opportunities for improvement, and to engage broadly their university community
in the assurance of quality.

...but the current system of governance faces three main challenges

Nonetheless, there are three fundamental challenges facing the institutions of quality
assurance that merit attention and improvement. First, the design of quality assurance
institutions creates conflicting responsibilities for the Ministry of Education. MEC
establishes, funds, and steers the federal university system, through its Secretariat for
Higher Education (SESu). At the same time, it is responsible, through SERES and,
indirectly, INEP, for evaluating their performance and for regulatory actions concerning
the programmes they offer. These conflicting responsibilities lead the nation’s higher
education institutions, especially its private institutions, to view the Ministry as a champion
of one sector rather than a neutral arbiter among all. As one representative of a private
higher education institution told to the review team, “For the people at MEC, the federal
universities are their children, and we are their bastards.”

Second, while CONAES is responsible for providing guidance and feedback on the
functioning of SINAES, it is not properly resourced and organised to do so. CONAES does
not have its own professional staff or a dedicated budget, and lacks a capacity to undertake
the sort of detailed and sustained analytical work that is needed to evaluate how SINAES
is working. Instead, it depends upon the input of implementing bodies whose work it is to
supervise and guide, most especially INEP. This dependence is exacerbated by the
participation of the implementing bodies on the council itself. It lacks sufficiently wide

172 | CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE

input — from professional bodies, employer associations, and other centres of government
- to take into account the broader social responsibilities of higher education.

Finally, in most higher education systems, responsibility for promoting and sharing quality
improvement practices lies with bodies outside of government - with associations that
represent sub-sectors (such as research, confessional, or polytechnic universities), and with
bodies that represent professional groups within higher education institutions, including
institutional research, curriculum design, assessment, and quality assurance. Examples of
such bodies with such a role in other OECD countries include the German Rectors’
Conference (HRK, 2018;4)), Universities UK (UUK, 2018;5)) and the National Association
of Universities and Higher Education Institutions in Mexico (ANUIES, 2018{6)). The
review found few examples of the engagement of equivalent bodies in Brazil in research,
advocacy, and training in support of quality improvement, and little attention on the part of
public authorities to their potentially important role.

In addition to these three structural issues, there is the question of the coverage of the
federal quality assurance system. As noted earlier, the systems for external quality
assurance of HEIs and undergraduate programmes analysed in this report apply only to
private HEIs and federal public HEIs. State and municipal public institutions — which
account for almost 10% of enrolment - are not subject to SINAES, but rather to state-level
regulatory and quality assurance rules. Although this situation reflects the constitutional
distribution of competences in the Brazilian state, which allows considerable autonomy to
states and municipalities, it leads to a fragmented system and means there is no single
national benchmark of higher education quality. A single quality reference framework
would make external quality assurance for higher education more transparent and
understandable for students and their families.

8.3. Key recommendations concerning governance

To respond to the challenges outlined above, we believe that four related policy choices
merit consideration.

1. Create an independent quality assurance agency

The work to design and create any new agency for quality assurance in Brazil will need to
address some key questions:

CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE | 173

units for different sets of disciplines (natural sciences, social sciences etc.) should be
considered. Such units, integrated within the agency, could potentially allow evaluation to
be better tailored to individual disciplines and work more closely with the discipline-
specific CAPES evaluations.

How should the new agency be funded? The current system of external quality assurance
in Brazil is funded by a combination of public resources (paying the salaries of public
servants, for example) and fees paid by institutions for evaluation activities. Quality
assurance agencies in a number of systems, including the Portuguese example mentioned
above, are funded primarily through fees from institutions. To ensure efficient use of public
resources, this should be the long-term aim in Brazil. A thorough analysis will be required
to determine the costs of a new agency and the level of fees needed to finance its operation.

2. Strengthen CONAES

To ensure that the quality assurance agency has an advisory council that brings a wide
social vision to its work, CONAES could take on this responsibility, after substantially
modification. CONAES would be a council holding fixed and staggered terms to ensure
their independence of government, and encompass balanced representation from students,
public and private sector employers, instructors from public and private higher education
institutions, higher education administrators, leading researchers, and the senior policy
official in MEC with responsibility for taking a comprehensive view of higher education.

3. Restructure the government departments that are responsible for higher
education

174 | CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE

and strategic view of the entire Brazilian higher education system — which the Ministry
presently lacks. Units organised along sectoral lines, for example, could support the work
of a senior official. These might include groups responsible for (a) federal universities; (b)
private universities; (c) technical higher education; and (d) coordination with state and
municipal higher education institutions. This scheme of organisation would benefit the
nation’s quality assurance system by supporting a strategic and comprehensive vision for
the higher education system, by clarifying the role of private provision within the system,
and by encouraging continued differentiation of institutions and policies.

4. Incentivise the development of expertise in quality assurance in sector
organisations

5. Explore how a reformed external quality assurance system could also apply
to state and municipal institutions

Notes

' CAPES regulates all academic postgraduate provision in Brazil, including in state and municipal
public universities and university centres. SERES and INEP only regulate institution in the Federal
higher education system.

CHAPTER 8. GOVERNANCE OF EXTERNAL QUALITY ASSURANCE | 175

References

ANUIES (2018), Asociacion Nacional de Universidades e Instituciones de Educacion Superior,
Asociacion Nacional de Universidades e Instituciones de Educacion Superior,
http://www.anuies.mx/# (accessed on 18 November 2018).

CHEA (2016), The CIOG International Quality Principles: Toward a Shared Understanding of
Quality, Council for Higher Education Accreditation , Washington, DC, http://www.chea..

Congreso Nacional (2012), Projeto de lei: cria o Instituto Nacional de Supervisdo e Avaliagado
da Educagao Superior - INSAES, e da outras providéncias., Secretaria de Assuntos
Parlamentares, http://www.planalto.gov.br/ccivil_03/Projetos/PL/2012/msg398-
3lagosto2012.htm (accessed on 30 November 2018).

ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education
Area (ESG).

HRK (2018), The HRK - German Rectors' Conference, https://www.hrk.de/hrk-at-a-glance/
(accessed on 18 November 2018).

INQAAHE (2016), INOAAHE Guidelines of Good Practices 2016 - revised edition, International
Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona,
http://www.ingaahe.org/.

UUK (2018), Universities UK, https://www.universitiesuk.ac.uk/ (accessed on
18 November 2018).

[6]

[3]

[7]

[1]

[4]

[2]

[5]

ANNEX A. REVIEW TEAM. | 177

Annex A. Review team

Maria José Lemaitre is Executive Director of the Centro Interuniversitario de Desarrollo
(CINDA), an international organisation dedicated to promoting effective higher education
policy, based in Santiago, Chile. A sociologist and education specialist, Maria José has
been responsible for the development and implementation of quality assurance processes
in higher education in Chile. She has chaired the Accreditation Agencies International
Network (INQAAHE) and the Iberoamerican Network for Quality Assurance (RIACES)
and has served in the Advisory Council of the Spanish Quality Assurance agency (ANECA)
and in the International Commission for the Council for Higher Education Accreditation of
the United States (CHEA). She has provided advice on higher education and quality
assurance issues to governments, higher education institutions and bilateral and multilateral
organisations in many regions of the world.

Pedro N. Teixeira is Associate Professor at the Department of Economics (University of
Porto) and Director of CIPES (Centre of Research on Higher Education Policy). He was
previously Vice Rector at the University of Porto, responsible for the area of Academic
Affairs and Teaching and Learning. Pedro is special adviser on higher education to the
President of Portugal and a member of the evaluation panels for the Institutional Evaluation
Programme of the European University Association (EUA) and the Reviews of the
European Association for Quality Assurance in Higher Education (ENQA). He holds a PhD
in Economics (Exeter University, UK), a masters in the Economics of Higher Education at
CHEPS (University of Twente, The Netherlands), and a bachelor’s degree in Economics
(University of Porto). His research interests focus on the economics of higher education,
notably on markets and privatisation, and the development of human capital as a research
programme.

Thomas Weko is a senior analyst at the OECD specialising in higher education. He
coordinated the review. He worked on OECD’s Tertiary Education for the Knowledge
Society in 2005-2006, and subsequently served in the US Department of Education, first as
Associate Commissioner for Postsecondary, Adult, and Career Education at the National
Center for Education Statistics, and then as Director of the Policy and Program Studies
Service. He previously worked at the Washington State Higher Education Coordinating
Board, at the US Government Accountability Office, and as a university professor.

Simon Roy is an analyst at the OECD, with a background in international higher education
policy. He joined the OECD in 2017 from the European Commission, where he was a
policy officer in DG Education and Culture's higher education policy team. During his time
at the Commission, Simon worked on analysis and policy cooperation activities involving
EU Member States and partner countries, including in the fields of higher education
funding, performance measurement and graduate skills and employability. He coordinated
preparation of the Commission's most recent Communication on higher education, focused
on EU cooperation in the field and published in May 2017.

178 | ANNEX B. SCHEDULE FOR THE REVIEW TEAM’S MISSION TO BRAZIL

Annex B. Schedule for the review team’s mission to Brazil

Monday, 5 March, Brasilia

9:00-10:00 Ministry of Education (MEC)
e Executive Secretary (Vice-Minister)
10:15-11:15 MEC - Secretariat for Higher Education (SESU)
e = Secretary

e General Coordinator for Legislation on Higher Education
e Director for the Development of Federal Institutions of HE
e — General Coordinator for Higher Education Policy
11:30 - 12:30 MEC - VET Secretary (SETEC)
e Secretary
e Director of VET Policy and Regulation
e General Coordinator of VET Regulation and Supervision
13:30 - 14:30 MEC - HE Regulation and Supervision Secretary (SERES)
e = Secretary
e Director for Higher Education Supervision
e Director for Regulatory Policy
e — Director for Higher Education Regulation

14:45 - 15:45 Science and Technology Development National Council (CNPq)
e President

16:00 - 16:50 Anisio Teixeira National Institute for Education Research and Innovation (INEP)
e President

17:00 - 18:00 INEP

e Director for Higher Education Assessment
Tuesday, 6 March, Brasilia

8:00 - 8:40 INEP - Enade General Coordination (CGENADE)
e — General Coordinator for ENADE
8:50 - 9:30 INEP — General Coordination of Higher Education Institutions Assessment (CGACGIES)
e — General Coordinator
9:40 -10:20 INEP — General Coordination for Quality Control in Higher Education (CGCQES)
e — General Coordinator
10:30 - 12:30 National Commission for Higher Education Assessment (CONAES)
e President and Members
14:30 - 15:00 Graduate Education Coordination (CAPES)
e President
15:00 - 16:00 CAPES — Assessment Directorate
e Director
16:00 - 16:40 CAPES - General Coordination for Assessment and Monitoring
e Coordinator
16:40 - 17:20 CAPES — General Coordination for Support to Graduate Activities Coordinator (CGAP)
e Coordinator
17:20 - 18:00 CAPES - General Coordination to Monitor and Assess Professional Masters (CGNE)

e = Coordinator

ANNEX B. SCHEDULE FOR THE REVIEW TEAM’S MISSION TO BRAZIL | 179

Wednesday, 7 March, Brasilia

10:00 - 11:30
11:45 - 12:30

14:00 - 15:00
15:00 - 17:40

18:00 - 18:40

Thursday, 8 March, Recife

9:00 - 12:10

14:30 - 18:00

Friday, 9 March, Recife
9:00 - 10:00

10:15 - 12:30

15:15 - 17:30

University Centre of Brasilia (UniCEUB)
e — Senior administration
Northern University of Parana (Unopar)
National Association of Directors of Federal Institutions of Higher Education (ANDIFES)
University of Brasilia (UnB)
e — Senior Administration
e Teaching staff
Association of Private Higher Education Institutions (ANUP)

Federal University of Pernambuco (UFPE)
e — Senior Administration
e — Teaching staff
e — Students
Mauricio de Nassau University Centre (UNINASSAU)
e — Senior Administration
e Teaching staff
e — Students

Association of the owners of private institutions (ABMES)
e — President
e Vice-president
Federal Institute for Education, Science and Technology of Pernambuco (IFPE)
e — Senior Administration
e Teaching staff
Catholic University of Pernambuco (UNICAP)
e — Senior Administration
e Teaching staff

Monday, 12 March, Sao Paulo

9:00 - 11:30

14:00 - 15:00

15:10 - 16:30
16:45 — 18:00

University of Sao Paulo (USP)
e Senior Administration
e Teaching staff
e Students
The Foundation for Research Support of the State of Sao Paulo (FAPESP)
e President, Director for the Technical and Administrative Council
e — Scientific Director
Sao Paulo Industry Association School (SENAI)
Private University Association for the Development of Higher Education (ABRAES)
e — President Executive Director

Tuesday, 13 March, Sao Paulo

9:00 - 12:20

14:00 - 15:00
15:15 - 16:45
17:00 - 18:30

Anhanguera University/Kroton
e — Senior management
e Teaching staff
Association of private institutions in Sao Paulo (SEMESP)
Getulio Vargas Foundation - FGV
Municipal University of Sao Caetano do Sul (USCS)
e — Senior management

Wednesday, 14 March, Sao Paulo

9:00 — 10:00

10:10 - 11:00

11:10 - 12:00

Brazilian Bar Association (OAB)
e Representative

Brazilian Society for the Advancement of Science (SBPC)
e Representatives

National Association of Graduate Studies and Research in Social Sciences (ANPOCS)
e Representatives

ORGANISATION FOR EGONOMIC CO-OPERATION
AND DEVELOPMENT

The OECD is a unique forum where governments work together to address the economic, social and
environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand and to
help governments respond to new developments and concerns, such as corporate governance, the
information economy and the challenges of an ageing population. The Organisation provides a setting
where governments can compare policy experiences, seek answers to common problems, identify good
practice and work to co-ordinate domestic and international policies.

The OECD member countries are: Australia, Austria, Belgium, Canada, Chile, the Czech Republic,
Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Korea,
Latvia, Lithuania, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the
Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States.
The European Union takes part in the work of the OECD.

OECD Publishing disseminates widely the results of the Organisation’s statistics gathering and
research on economic, social and environmental issues, as well as the conventions, guidelines and
standards agreed by its members.

OECD PUBLISHING, 2, rue André-Pascal, 75775 PARIS CEDEX 16
(91 2018 18 1 P) ISBN 978-92-64-30904-3- 2018

Reviews of National Policies for Education

Rethinking Quality Assurance for Higher Education
in Brazil

This review examines the external systems in place to assure the quality of higher education in Brazil. It
highlights the relative success of the Brazilian quality assurance model in regulating market entry for private
operators in Brazil, which cater to over 70% of students. But it also calls into question the effectiveness of
existing systems to monitor the quality of undergraduate programmes and institutions and the ability of public
authorities to act decisively to protect students from bad quality education.

The report analyses the systems that regulate the launch of new higher education institutions and programmes
and evaluates quality assessment mechanisms for existing programmes and institutions. It also looks at the
structures allowing public authorities to intervene to end or improve poor quality provision.

The review offers policy recommendations. It proposes a more differentiated system of quality assurance. It
also recommends significant modifications to the design and purpose of the National Examination of Student
Performance (ENADE).

This work was requested by the National Commission for Evaluation of Higher Education (CONAES) and
CAPES, the federal body responsible for quality assurance and funding of postgraduate education.

Consult this publication on line at https://doi.org/10.1787/9789264309050-en.

This work is published on the OECD iLibrary, which gathers all OECD books, periodicals and statistical databases.
Visit www.oecd-ilibrary.org for more information.

ISBN 978-92-64-30904-3
91 2018 18 1 P

OECD publishing

www.oecd.org/publishing

89264

509045

Internet Archive Audio

Featured

Top

Images

Featured

Top

Software

Featured

Top

Texts

Featured

Top

Video

Featured

Top

Mobile Apps

Browser Extensions

Archive-It Subscription

Save Page Now

Full text of "Rethinking Quality Assurance For Higher Educarion in Brazil"

See other formats